{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "99d3711b-99c4-43b2-bb43-7f2793b35c3f",
   "metadata": {},
   "source": [
    "# 神经网络快速入门与实践"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4da47d9-50a7-43d0-aaa8-438c1e21b0df",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## 基于NumPy的神经网络构建与训练"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ebe49c8-8388-4f31-b922-21bc56e9774a",
   "metadata": {},
   "source": [
    "### 示例1：最简单的神经网络模型（y=wx+b）\n",
    "- 单个神经元构成的单层网络（仅有一个输出层）\n",
    "- 输入特征为1\n",
    "- 输入样本数量为1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95be273d-0f09-45a9-86e8-4113d9f88d41",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# 数据\n",
    "x = 2.0   # 输入，\n",
    "y = 4.0   # 真实值\n",
    "\n",
    "# 参数初始化\n",
    "w = 1.0\n",
    "b = 0.0\n",
    "lr = 0.1\n",
    "\n",
    "# 前向传播\n",
    "y_hat = w * x + b\n",
    "loss = 0.5 * (y_hat - y) ** 2\n",
    "\n",
    "# 反向传播\n",
    "dL_dyhat = y_hat - y\n",
    "dL_dw = dL_dyhat * x\n",
    "dL_db = dL_dyhat * 1\n",
    "\n",
    "# 参数更新\n",
    "w = w - lr * dL_dw\n",
    "b = b - lr * dL_db\n",
    "\n",
    "print(f\"Updated w: {w}, b: {b}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "752a6008-807c-4e89-a50a-57d8505c60fa",
   "metadata": {},
   "source": [
    "### 示例2：单层网络，输入为矩阵形式（y=Xw+b）\n",
    "- 扩展前面的示例：\n",
    "  - 输入特征为2\n",
    "  - 样本数量为2，每个样本2个特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15ddb448-5711-406a-acd9-deda2fbc487e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# 模拟数据\n",
    "X = np.array([[1.0, 2.0], [3.0, 4.0]])  # shape: (2, 2)，输入特征为2，样本数量为2\n",
    "y = np.array([[5.0], [11.0]])          # shape: (2, 1)，每个样本都有一个真实值\n",
    "\n",
    "# 初始化参数\n",
    "# 两个输入特征的各初始权重，因为只有一个神经元，\n",
    "# 这里的权重矩阵表现为列向量的形式\n",
    "b = 0.0\n",
    "w = np.array([[0.1], [0.1]])           # shape: (2, 1)\n",
    "lr = 0.01\n",
    "N = X.shape[0]\n",
    "\n",
    "# 前向传播\n",
    "y_hat = X @ w + b\n",
    "loss = 0.5 * np.mean((y_hat - y) ** 2)\n",
    "\n",
    "# 反向传播\n",
    "grad_yhat = (y_hat - y) / N\n",
    "grad_w = X.T @ grad_yhat\n",
    "grad_b = np.sum(grad_yhat)\n",
    "\n",
    "# 更新参数\n",
    "w -= lr * grad_w\n",
    "b -= lr * grad_b\n",
    "\n",
    "print(\"Updated w:\", w.ravel())\n",
    "print(\"Updated b:\", b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e6e41ec-5718-4e94-8407-8a9d7fd068fb",
   "metadata": {},
   "source": [
    "### 示例3：两层网络示例（y=XW+b）\n",
    "- 两层网络：一个隐藏层，一个输出层\n",
    "- 样本数为N=2，输入特征维度为D=2\n",
    "- 隐藏层神经元数H=4，使用ReLU激活函数\n",
    "- 损失函数：均方误差（MSE）\n",
    "  - $L = \\frac{1}{2N} \\sum_{i=1}^N (\\hat{y}_i - y_i)^2$\n",
    "\n",
    "**神经网络结构：**\n",
    "```mathematica\n",
    "输入 X ∈ ℝ^{N×D}\n",
    "   ↓\n",
    "第一层（线性）: Z₁ = X @ W₁ + b₁      # shape: (N, H)\n",
    "   ↓\n",
    "ReLU 激活函数: A₁ = ReLU(Z₁)           # shape: (N, H)\n",
    "   ↓\n",
    "第二层（线性）: ŷ = A₁ @ W₂ + b₂      # shape: (N, 1)\n",
    "   ↓\n",
    "MSE 损失函数: L = MSE(ŷ, y)\n",
    "```\n",
    "\n",
    "其中的符号表示：\n",
    "- 𝐷：输入特征数\n",
    "- 𝐻：隐藏层维度（神经元数）\n",
    "- 𝑁：样本数\n",
    "- ReLU激活：ReLU(𝑧)=max(0,𝑧)\n",
    "\n",
    "**前向传播与反向传播流程图**\n",
    "```scss\n",
    "前向传播：\n",
    "X → Linear(W1, b1) → Z1 → ReLU → A1 → Linear(W2, b2) → y_hat → MSE → Loss\n",
    "\n",
    "反向传播（链式）：\n",
    "Loss → d(y_hat) → dW2, db2 → dA1 → dZ1 (ReLU) → dW1, db1\n",
    "```\n",
    "\n",
    "#### 代码解析\n",
    "##### 1. 前向传播\n",
    "首先明确前向传播的计算过程，以便后续推导梯度。代码中的前向传播步骤如下：\n",
    "\n",
    "（1）隐藏层计算：$Z_1 = X W_1 + b_1$\n",
    "- $ X \\in \\mathbb{R}^{N \\times D} $，$ W_1 \\in \\mathbb{R}^{D \\times H} $，$ b_1 \\in \\mathbb{R}^{1 \\times H} $。\n",
    "- $ Z_1 \\in \\mathbb{R}^{N \\times H} $ 是隐藏层的线性输出。\n",
    "- 代码实现\n",
    "  ```python\n",
    "  Z1 = X @ W1 + b1  # (N x H)\n",
    "  ```\n",
    "（2）ReLU激活：$A_1 = \\text{ReLU}(Z_1) = \\max(0, Z_1)$\n",
    "- ReLU 逐元素操作：如果  $ Z_1_{ij} > 0 $，则 A_1_{ij} = Z_1_{ij} ，否则 A_1_{ij} = 0 。\n",
    "- $ A_1 \\in \\mathbb{R}^{N \\times H} $。\n",
    "- 代码实现\n",
    "  ```python\n",
    "  A1 = np.maximum(0, Z1)  # (N x H)\n",
    "  ```\n",
    "\n",
    "（3）输出层计算：$\\hat{y} = A_1 W_2 + b_2$\n",
    "- $ A_1 \\in \\mathbb{R}^{N \\times H} $，$ W_2 \\in \\mathbb{R}^{H \\times 1} $，$ b_2 \\in \\mathbb{R}^{1 \\times 1} $。\n",
    "- $ \\hat{y} \\in \\mathbb{R}^{N \\times 1} $。\n",
    "- 代码实现\n",
    "  ```python\n",
    "  y_hat = A1 @ W2 + b2  # (N x 1)\n",
    "  ```\n",
    "\n",
    "（4）损失函数：$L = \\frac{1}{2N} \\sum_{i=1}^N (\\hat{y}_i - y_i)^2$\n",
    "- 代码计算均方误差的平均值\n",
    "  ```python\n",
    "  loss = 0.5 * np.mean((y_hat - y) ** 2)\n",
    "  ```\n",
    "\n",
    "##### 2. 反向传播\n",
    "反向传播的目标是计算损失 $ L $ 对参数 $ W_1, b_1, W_2, b_2 $ 的梯度。我们使用链式法则，从输出层向输入层逐步推导。\n",
    "\n",
    "（1）**对损失函数的导数**\n",
    "\n",
    "对 $ \\hat{y} $ 求导：$\\frac{\\partial L}{\\partial \\hat{y}_i} = \\frac{1}{N} (\\hat{y}_i - y_i)$\n",
    "\n",
    "矩阵形式：$\\frac{\\partial L}{\\partial \\hat{y}} = \\frac{1}{N} (\\hat{y} - y) \\in \\mathbb{R}^{N \\times 1}$\n",
    "\n",
    "代码实现：\n",
    "```python\n",
    "dy_hat = (y_hat - y) / N  # (N x 1)\n",
    "```\n",
    "\n",
    "（2）**输出层参数梯度 ($ W_2, b_2 $)**\n",
    "\n",
    "输出层计算为：$\\hat{y} = A_1 W_2 + b_2$\n",
    "\n",
    "梯度 $ \\frac{\\partial L}{\\partial W_2} $\n",
    "- 根据链式法则：$\\frac{\\partial L}{\\partial W_2} = \\frac{\\partial L}{\\partial \\hat{y}} \\cdot \\frac{\\partial \\hat{y}}{\\partial W_2}$\n",
    "- 其中：$\\frac{\\partial \\hat{y}}{\\partial W_2} = A_1$\n",
    "\n",
    "因此：$\\frac{\\partial L}{\\partial W_2} = A_1^T \\cdot \\frac{\\partial L}{\\partial \\hat{y}}$\n",
    "- $ A_1^T \\in \\mathbb{R}^{H \\times N} $，$ \\frac{\\partial L}{\\partial \\hat{y}} \\in \\mathbb{R}^{N \\times 1} $。\n",
    "- 结果：$ \\frac{\\partial L}{\\partial W_2} \\in \\mathbb{R}^{H \\times 1} $。\n",
    "\n",
    "代码实现：\n",
    "```python\n",
    "dW2 = A1.T @ dy_hat  # (H x 1)\n",
    "```\n",
    "\n",
    "梯度$ \\frac{\\partial L}{\\partial b_2} $：\n",
    "- 根据链式法则：$\\frac{\\partial L}{\\partial b_2} = \\frac{\\partial L}{\\partial \\hat{y}} \\cdot \\frac{\\partial \\hat{y}}{\\partial b_2}$\n",
    "- 其中：$\\frac{\\partial \\hat{y}}{\\partial b_2} = 1$\n",
    "\n",
    "因此：$\\frac{\\partial L}{\\partial b_2} = \\text{sum}(\\frac{\\partial L}{\\partial \\hat{y}}, \\text{axis}=0)$\n",
    "- 对 $ \\frac{\\partial L}{\\partial \\hat{y}} \\in \\mathbb{R}^{N \\times 1} $ 按行求和，保持维度为 $ (1 \\times 1) $。\n",
    "\n",
    "代码实现：\n",
    "```python\n",
    "db2 = np.sum(dy_hat, axis=0, keepdims=True)  # (1 x 1)\n",
    "```\n",
    "\n",
    "（3）**传播到隐藏层**\n",
    "\n",
    "需要计算 $ \\frac{\\partial L}{\\partial A_1} $ 和 $ \\frac{\\partial L}{\\partial Z_1} $，以便进一步推导 $ W_1, b_1 $ 的梯度。\n",
    "\n",
    "梯度 $ \\frac{\\partial L}{\\partial A_1} $：\n",
    "- 根据链式法则：$\\frac{\\partial L}{\\partial A_1} = \\frac{\\partial L}{\\partial \\hat{y}} \\cdot \\frac{\\partial \\hat{y}}{\\partial A_1}$\n",
    "- 其中：$\\hat{y} = A_1 W_2 + b_2$， $\\frac{\\partial \\hat{y}}{\\partial A_1} = W_2$\n",
    "\n",
    "因此：$\\frac{\\partial L}{\\partial A_1} = \\frac{\\partial L}{\\partial \\hat{y}} \\cdot W_2^T$\n",
    "- $ \\frac{\\partial L}{\\partial \\hat{y}} \\in \\mathbb{R}^{N \\times 1} $，$ W_2^T \\in \\mathbb{R}^{1 \\times H} $。\n",
    "- 结果：$ \\frac{\\partial L}{\\partial A_1} \\in \\mathbb{R}^{N \\times H} $。\n",
    "\n",
    "代码实现：\n",
    "```python\n",
    "dA1 = dy_hat @ W2.T  # (N x H)\n",
    "```\n",
    "\n",
    "梯度 $ \\frac{\\partial L}{\\partial Z_1} $：\n",
    "- 隐藏层使用ReLU激活函数，$A_1 = \\text{ReLU}(Z_1) = \\max(0, Z_1)$\n",
    "- ReLU的导数为：$\\frac{\\partial A_1}{\\partial Z_1} = \\begin{cases} \n",
    "1, & \\text{if } Z_1 > 0 \\\\\n",
    "0, & \\text{if } Z_1 \\leq 0 \n",
    "\\end{cases}$\n",
    "\n",
    "因此：$\\frac{\\partial L}{\\partial Z_1} = \\frac{\\partial L}{\\partial A_1} \\cdot \\frac{\\partial A_1}{\\partial Z_1}$\n",
    "- $ \\frac{\\partial A_1}{\\partial Z_1} $ 是一个逐元素操作，值为 1（当 $ Z_1 > 0 $）或 0（当 $ Z_1 \\leq 0 $）。\n",
    "- 代码中用 (Z1 > 0).astype(float) 实现ReLU的导数。\n",
    "- 结果：$ \\frac{\\partial L}{\\partial Z_1} \\in \\mathbb{R}^{N \\times H} $。\n",
    "\n",
    "代码实现：\n",
    "```python\n",
    "dZ1 = dA1 * (Z1 > 0).astype(float)  # (N x H)\n",
    "```\n",
    "\n",
    "（4）**隐藏层参数梯度 ($ W_1, b_1 $)**\n",
    "\n",
    "隐藏层计算为：$Z_1 = X W_1 + b_1$\n",
    "\n",
    "梯度 $ \\frac{\\partial L}{\\partial W_1} $\n",
    "- 根据链式法则：$\\frac{\\partial L}{\\partial W_1} = \\frac{\\partial L}{\\partial Z_1} \\cdot \\frac{\\partial Z_1}{\\partial W_1}$\n",
    "- 其中：$\\frac{\\partial Z_1}{\\partial W_1} = X$\n",
    "\n",
    "因此：$\\frac{\\partial L}{\\partial W_1} = X^T \\cdot \\frac{\\partial L}{\\partial Z_1}$\n",
    "- $ X^T \\in \\mathbb{R}^{D \\times N} $，$ \\frac{\\partial L}{\\partial Z_1} \\in \\mathbb{R}^{N \\times H} $。\n",
    "- 结果：$ \\frac{\\partial L}{\\partial W_1} \\in \\mathbb{R}^{D \\times H} $。\n",
    "\n",
    "代码实现：\n",
    "```python\n",
    "dW1 = X.T @ dZ1  # (D x H)\n",
    "```\n",
    "\n",
    "梯度 $ \\frac{\\partial L}{\\partial b_1} $：\n",
    "- 根据链式法则：$\\frac{\\partial L}{\\partial b_1} = \\frac{\\partial L}{\\partial Z_1} \\cdot \\frac{\\partial Z_1}{\\partial b_1}$\n",
    "- 其中：$\\frac{\\partial Z_1}{\\partial b_1} = 1$\n",
    "\n",
    "因此：$\\frac{\\partial L}{\\partial b_1} = \\sum_{i=1}^N \\frac{\\partial L}{\\partial Z_{1,i}}$\n",
    "- 对 $ \\frac{\\partial L}{\\partial Z_1} \\in \\mathbb{R}^{N \\times H} $ 按行求和，保持维度为 $ (1 \\times H) $。\n",
    "\n",
    "代码实现：\n",
    "```python\n",
    "db1 = np.sum(dZ1, axis=0, keepdims=True)  # (1 x H)\n",
    "```\n",
    "\n",
    "##### 3. 参数更新\n",
    "使用梯度下降更新参数：$\\theta \\gets \\theta - \\eta \\cdot \\frac{\\partial L}{\\partial \\theta}$\n",
    "\n",
    "其中 $ \\eta = 0.01 $ 是学习率，$ \\theta $ 代表 $ W_1, b_1, W_2, b_2 $。\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51ab0a69-14b5-4e4a-8076-fe91ded7b54a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# ---------- 初始化 ----------\n",
    "np.random.seed(42)\n",
    "\n",
    "# 输入数据 X 和目标 y\n",
    "X = np.array([[1.0, 2.0], [3.0, 4.0]])  # shape: (2, 2)\n",
    "y = np.array([[5.0], [11.0]])          # shape: (2, 1)\n",
    "N, D = X.shape\n",
    "H = 4  # 隐藏层单元数\n",
    "lr = 0.01\n",
    "\n",
    "# 权重初始化\n",
    "W1 = np.random.randn(D, H) * 0.01     # 第一层权重 (D, H)\n",
    "b1 = np.zeros((1, H))                 # 第一层偏置 (1, H)\n",
    "W2 = np.random.randn(H, 1) * 0.01     # 第二层权重 (H, 1)\n",
    "b2 = np.zeros((1, 1)) \n",
    "\n",
    "# ---------- 前向传播 ----------\n",
    "Z1 = X @ W1 + b1            # (N x H)\n",
    "A1 = np.maximum(0, Z1)      # ReLU 激活 → (N x H)\n",
    "y_hat = A1 @ W2 + b2        # 输出 → (N x 1)\n",
    "loss = 0.5 * np.mean((y_hat - y) ** 2)\n",
    "\n",
    "# ---------- 反向传播 ----------\n",
    "dy_hat = (y_hat - y) / N                # dL/dy_hat → (N x 1)\n",
    "\n",
    "# 输出层参数梯度\n",
    "dW2 = A1.T @ dy_hat                     # (H x 1)\n",
    "db2 = np.sum(dy_hat, axis=0, keepdims=True)  # (1 x 1)\n",
    "\n",
    "# 传播到隐藏层\n",
    "dA1 = dy_hat @ W2.T                     # (N x H)\n",
    "dZ1 = dA1 * (Z1 > 0).astype(float)      # ReLU 导数 → (N x H)\n",
    "\n",
    "# 隐藏层参数梯度\n",
    "dW1 = X.T @ dZ1                         # (D x H)\n",
    "db1 = np.sum(dZ1, axis=0, keepdims=True)  # (1 x H)\n",
    "\n",
    "# ---------- 参数更新 ----------\n",
    "W1 -= lr * dW1\n",
    "b1 -= lr * db1\n",
    "W2 -= lr * dW2\n",
    "b2 -= lr * db2\n",
    "\n",
    "# ---------- 输出 ----------\n",
    "print(\"Loss:\", loss)\n",
    "print(\"Updated W1:\\n\", W1)\n",
    "print(\"Updated b1:\\n\", b1)\n",
    "print(\"Updated W2:\\n\", W2)\n",
    "print(\"Updated b2:\\n\", b2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dae34488-21dd-46ed-aded-00f916744cea",
   "metadata": {},
   "source": [
    "### 扩展以上网络，使用两个隐藏层的示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37e3ce42-1d14-4b59-b9e3-438b9fcd12c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# 设置随机种子\n",
    "np.random.seed(42)\n",
    "\n",
    "# ---------- 模拟数据 ----------\n",
    "X = np.array([[1.0, 2.0], [3.0, 4.0]])  # shape: (2, 2)\n",
    "y = np.array([[5.0], [11.0]])          # shape: (2, 1)\n",
    "N, D = X.shape\n",
    "\n",
    "# 网络结构参数\n",
    "H1 = 4  # 第 1 个隐藏层神经元个数\n",
    "H2 = 4  # 第 2 个隐藏层神经元个数\n",
    "lr = 0.01\n",
    "\n",
    "# ---------- 初始化参数 ----------\n",
    "W1 = np.random.randn(D, H1) * np.sqrt(2. / D)\n",
    "b1 = np.zeros((1, H1))\n",
    "\n",
    "W2 = np.random.randn(H1, H2) * np.sqrt(2. / H1)\n",
    "b2 = np.zeros((1, H2))\n",
    "\n",
    "W3 = np.random.randn(H2, 1) * np.sqrt(2. / H2)\n",
    "b3 = np.zeros((1, 1))\n",
    "\n",
    "# ---------- 前向传播 ----------\n",
    "Z1 = X @ W1 + b1         # (N, H1)\n",
    "A1 = np.maximum(0, Z1)   # ReLU → (N, H1)\n",
    "\n",
    "Z2 = A1 @ W2 + b2        # (N, H2)\n",
    "A2 = np.maximum(0, Z2)   # ReLU → (N, H2)\n",
    "\n",
    "y_hat = A2 @ W3 + b3     # (N, 1)\n",
    "\n",
    "loss = 0.5 * np.mean((y_hat - y) ** 2)\n",
    "\n",
    "# ---------- 反向传播 ----------\n",
    "dy_hat = (y_hat - y) / N   # (N, 1)\n",
    "\n",
    "# 输出层梯度\n",
    "dW3 = A2.T @ dy_hat        # (H2, 1)\n",
    "db3 = np.sum(dy_hat, axis=0, keepdims=True)  # (1, 1)\n",
    "\n",
    "# 第二隐藏层梯度\n",
    "dA2 = dy_hat @ W3.T        # (N, H2)\n",
    "dZ2 = dA2 * (Z2 > 0)       # ReLU'(Z2)\n",
    "dW2 = A1.T @ dZ2           # (H1, H2)\n",
    "db2 = np.sum(dZ2, axis=0, keepdims=True)  # (1, H2)\n",
    "\n",
    "# 第一隐藏层梯度\n",
    "dA1 = dZ2 @ W2.T           # (N, H1)\n",
    "dZ1 = dA1 * (Z1 > 0)       # ReLU'(Z1)\n",
    "dW1 = X.T @ dZ1            # (D, H1)\n",
    "db1 = np.sum(dZ1, axis=0, keepdims=True)  # (1, H1)\n",
    "\n",
    "# ---------- 参数更新 ----------\n",
    "W3 -= lr * dW3\n",
    "b3 -= lr * db3\n",
    "W2 -= lr * dW2\n",
    "b2 -= lr * db2\n",
    "W1 -= lr * dW1\n",
    "b1 -= lr * db1\n",
    "\n",
    "# ---------- 输出 ----------\n",
    "print(\"Loss:\", loss)\n",
    "print(\"\\nUpdated W1:\\n\", W1)\n",
    "print(\"Updated b1:\\n\", b1)\n",
    "print(\"\\nUpdated W2:\\n\", W2)\n",
    "print(\"Updated b2:\\n\", b2)\n",
    "print(\"\\nUpdated W3:\\n\", W3)\n",
    "print(\"Updated b3:\\n\", b3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd2dd04b-dd9d-49c5-a9bb-f1adc3f901ec",
   "metadata": {},
   "source": [
    "### 为以上代码添加多轮训练循环机制\n",
    "- 输入维度扩展到了20，隐藏层维度扩展为50\n",
    "- 训练轮数通过超参数epochs进行控制\n",
    "\n",
    "**提示**：可通过手动调整如下代码中的epoch值（例如0.01 -> 0.02 -> 0.05 -> 0.1），来验证不同学习率下模型训练的收敛速度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba7f81c8-e0cb-4db1-8dad-24d54217387a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# 设置随机种子，确保可重复性\n",
    "np.random.seed(42)\n",
    "\n",
    "# ---------- 模拟数据 ----------\n",
    "N, D_in, H1, H2, D_out = 256, 20, 50, 50, 1  # 样本数量256，输入维度20，隐藏层同为50，输出维度为1\n",
    "\n",
    "# 输入 X 和真实标签 y，形状分别为 (N, D_in) 和 (N, 1)\n",
    "X = np.random.randn(N, D_in)\n",
    "y = np.random.randn(N, D_out)\n",
    "\n",
    "# ---------- 权重初始化 ----------\n",
    "W1 = np.random.randn(D_in, H1) * 0.01\n",
    "b1 = np.zeros((1, H1))\n",
    "\n",
    "W2 = np.random.randn(H1, H2) * 0.01\n",
    "b2 = np.zeros((1, H2))\n",
    "\n",
    "W3 = np.random.randn(H2, D_out) * 0.01\n",
    "b3 = np.zeros((1, D_out))\n",
    "\n",
    "# ---------- 超参数 ----------\n",
    "learning_rate = 0.05\n",
    "epochs = 500\n",
    "\n",
    "# ---------- 激活函数 ----------\n",
    "def relu(x):\n",
    "    return np.maximum(0, x)\n",
    "\n",
    "def relu_grad(x):\n",
    "    return (x > 0).astype(float)\n",
    "\n",
    "# ---------- 训练循环 ----------\n",
    "for epoch in range(epochs):\n",
    "    # ---- 前向传播 ----\n",
    "    z1 = X @ W1 + b1      # (N, H1)\n",
    "    a1 = relu(z1)         # (N, H1)\n",
    "\n",
    "    z2 = a1 @ W2 + b2     # (N, H2)\n",
    "    a2 = relu(z2)         # (N, H2)\n",
    "\n",
    "    z3 = a2 @ W3 + b3     # (N, D_out)\n",
    "    y_pred = z3           # 输出层无激活函数（线性输出）\n",
    "\n",
    "    # ---- 计算损失 ----\n",
    "    loss = np.mean((y_pred - y) ** 2)\n",
    "\n",
    "    if epoch % 50 == 0:\n",
    "        print(f\"Epoch {epoch:3d} | Loss: {loss:.6f}\")\n",
    "\n",
    "    # ---- 反向传播 ----\n",
    "    dloss = 2 * (y_pred - y) / N   # MSE 损失的导数\n",
    "\n",
    "    # 第三层梯度\n",
    "    dW3 = a2.T @ dloss             # (H2, D_out)\n",
    "    db3 = np.sum(dloss, axis=0, keepdims=True)  # (1, D_out)\n",
    "\n",
    "    # 第二层反传\n",
    "    da2 = dloss @ W3.T            # (N, H2)\n",
    "    dz2 = da2 * relu_grad(z2)     # (N, H2)\n",
    "\n",
    "    dW2 = a1.T @ dz2              # (H1, H2)\n",
    "    db2 = np.sum(dz2, axis=0, keepdims=True)\n",
    "\n",
    "    # 第一层反传\n",
    "    da1 = dz2 @ W2.T              # (N, H1)\n",
    "    dz1 = da1 * relu_grad(z1)     # (N, H1)\n",
    "\n",
    "    dW1 = X.T @ dz1               # (D_in, H1)\n",
    "    db1 = np.sum(dz1, axis=0, keepdims=True)\n",
    "\n",
    "    # ---- 参数更新 ----\n",
    "    W3 -= learning_rate * dW3\n",
    "    b3 -= learning_rate * db3\n",
    "\n",
    "    W2 -= learning_rate * dW2\n",
    "    b2 -= learning_rate * db2\n",
    "\n",
    "    W1 -= learning_rate * dW1\n",
    "    b1 -= learning_rate * db1\n",
    "\n",
    "# ---------- 最终损失 ----------\n",
    "print(f\"Final Loss: {loss:.6f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37336df0-2c83-463c-b257-e1eb7e4c39a7",
   "metadata": {},
   "source": [
    "### Mini-Batch机制：将样本按批次进行训练\n",
    "\n",
    "**原因**\n",
    "\n",
    "将样本分批次进行训练（mini-batch 训练）的主要原因包括：\n",
    "- 计算效率：降低内存需求，适合大型数据集，充分利用硬件并行性。\n",
    "- 梯度稳定性：提供比单一样本更稳定的梯度估计，同时比全批量更高效。\n",
    "- 泛化能力：通过随机打乱和批次处理，增强模型的鲁棒性和泛化能力。\n",
    "- 灵活性：可调整 batch 大小，适应不同规模数据集和硬件条件。\n",
    "- 更快收敛：更频繁的参数更新有助于快速探索参数空间。\n",
    "\n",
    "下面的示例代码中，mini-batch训练通过将$ N = 256 $个样本分成8个batch（每个32个样本），实现了高效且稳定的训练过程。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b409deb-7cda-4e40-8709-845d03f3dedf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# 设置随机种子，确保可重复性\n",
    "np.random.seed(42)\n",
    "\n",
    "# ---------- 模拟数据 ----------\n",
    "N, D_in, H1, H2, D_out = 256, 20, 50, 50, 1  # 样本数量256，输入维度20，隐藏层同为50，输出维度为1\n",
    "batch_size = 32  # 每个mini-batch的大小\n",
    "\n",
    "# 输入 X 和真实标签 y，形状分别为 (N, D_in) 和 (N, 1)\n",
    "X = np.random.randn(N, D_in)\n",
    "y = np.random.randn(N, D_out)\n",
    "\n",
    "# ---------- 权重初始化 ----------\n",
    "W1 = np.random.randn(D_in, H1) * 0.01\n",
    "b1 = np.zeros((1, H1))\n",
    "W2 = np.random.randn(H1, H2) * 0.01\n",
    "b2 = np.zeros((1, H2))\n",
    "W3 = np.random.randn(H2, D_out) * 0.01\n",
    "b3 = np.zeros((1, D_out))\n",
    "\n",
    "# ---------- 超参数 ----------\n",
    "learning_rate = 0.1\n",
    "epochs = 500\n",
    "num_batches = N // batch_size  # 计算 batch 数量\n",
    "\n",
    "# ---------- 激活函数 ----------\n",
    "def relu(x):\n",
    "    return np.maximum(0, x)\n",
    "\n",
    "def relu_grad(x):\n",
    "    return (x > 0).astype(float)\n",
    "\n",
    "# ---------- 训练循环 ----------\n",
    "for epoch in range(epochs):\n",
    "    # 打乱数据（每个epoch随机化样本顺序）\n",
    "    indices = np.random.permutation(N)\n",
    "    X_shuffled = X[indices]\n",
    "    y_shuffled = y[indices]\n",
    "    \n",
    "    # 初始化 epoch 总损失\n",
    "    epoch_loss = 0.01\n",
    "    \n",
    "    # 按mini-batch迭代\n",
    "    for i in range(num_batches):\n",
    "        # 获取当前 batch 的数据\n",
    "        start_idx = i * batch_size\n",
    "        end_idx = start_idx + batch_size\n",
    "        X_batch = X_shuffled[start_idx:end_idx]  # (batch_size, D_in)\n",
    "        y_batch = y_shuffled[start_idx:end_idx]  # (batch_size, D_out)\n",
    "        \n",
    "        # ---- 前向传播 ----\n",
    "        z1 = X_batch @ W1 + b1      # (batch_size, H1)\n",
    "        a1 = relu(z1)               # (batch_size, H1)\n",
    "        z2 = a1 @ W2 + b2           # (batch_size, H2)\n",
    "        a2 = relu(z2)               # (batch_size, H2)\n",
    "        z3 = a2 @ W3 + b3           # (batch_size, D_out)\n",
    "        y_pred = z3                 # 输出层无激活函数（线性输出）\n",
    "        \n",
    "        # ---- 计算损失 ----\n",
    "        batch_loss = np.mean((y_pred - y_batch) ** 2)\n",
    "        epoch_loss += batch_loss\n",
    "        \n",
    "        # ---- 反向传播 ----\n",
    "        dloss = 2 * (y_pred - y_batch) / batch_size  # MSE 损失的导数\n",
    "        \n",
    "        # 第三层梯度\n",
    "        dW3 = a2.T @ dloss          # (H2, D_out)\n",
    "        db3 = np.sum(dloss, axis=0, keepdims=True)  # (1, D_out)\n",
    "        \n",
    "        # 第二层反传\n",
    "        da2 = dloss @ W3.T          # (batch_size, H2)\n",
    "        dz2 = da2 * relu_grad(z2)   # (batch_size, H2)\n",
    "        \n",
    "        dW2 = a1.T @ dz2            # (H1, H2)\n",
    "        db2 = np.sum(dz2, axis=0, keepdims=True)\n",
    "        \n",
    "        # 第一层反传\n",
    "        da1 = dz2 @ W2.T            # (batch_size, H1)\n",
    "        dz1 = da1 * relu_grad(z1)   # (batch_size, H1)\n",
    "        \n",
    "        dW1 = X_batch.T @ dz1       # (D_in, H1)\n",
    "        db1 = np.sum(dz1, axis=0, keepdims=True)\n",
    "        \n",
    "        # ---- 参数更新 ----\n",
    "        W3 -= learning_rate * dW3\n",
    "        b3 -= learning_rate * db3\n",
    "        W2 -= learning_rate * dW2\n",
    "        b2 -= learning_rate * db2\n",
    "        W1 -= learning_rate * dW1\n",
    "        b1 -= learning_rate * db1\n",
    "    \n",
    "    # 计算平均 epoch 损失\n",
    "    epoch_loss /= num_batches\n",
    "    \n",
    "    # 每 50 个 epoch 输出损失\n",
    "    if epoch % 50 == 0:\n",
    "        print(f\"Epoch {epoch:3d} | Loss: {epoch_loss:.6f}\")\n",
    "\n",
    "# ---------- 最终损失（全数据集） ----------\n",
    "z1 = X @ W1 + b1\n",
    "a1 = relu(z1)\n",
    "z2 = a1 @ W2 + b2\n",
    "a2 = relu(z2)\n",
    "z3 = a2 @ W3 + b3\n",
    "y_pred = z3\n",
    "final_loss = np.mean((y_pred - y) ** 2)\n",
    "print(f\"Final Loss: {final_loss:.6f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c9a4c11-030a-4a06-9a4e-d2865f71fd30",
   "metadata": {},
   "source": [
    "### 正则化机制：降低过拟合风险\n",
    "正则化是神经网络训练中防止模型过拟合的核心机制，其核心思想是通过对模型施加约束，限制其复杂度，从而提升在未知数据上的泛化能力。\n",
    "\n",
    "**正则化的主要作用**：\n",
    "\n",
    "1. 抑制过拟合：减少模型对训练数据噪声的敏感度\n",
    "2. 提升泛化性：增强模型对新数据的适应能力\n",
    "3. 特征选择与稀疏化（L1）：将不重要特征的权重压缩至0，实现自动特征筛选\n",
    "4. 平滑权重分布（L2）：使所有权重趋近于较小值，避免极端权重\n",
    "5. 加速训练：简化模型结构，优化收敛效率\n",
    "\n",
    "**常用的正则化方法**：\n",
    "| 正则化方法             | 描述                                                                 | 公式表示                              | 特点                                                                 |\n",
    "|------------------------|----------------------------------------------------------------------|---------------------------------------|----------------------------------------------------------------------|\n",
    "| **L1正则化（Lasso）**  | 在损失函数中加入权重的L1范数惩罚项，鼓励模型产生稀疏权重              | $J(\\theta) = \\text{Loss} + \\lambda \\sum_{i} \\vert\\theta_i\\vert$ | 实现特征选择，产生稀疏模型，但可能导致解的不稳定性                |\n",
    "| **L2正则化（Ridge）**  | 在损失函数中加入权重的L2范数惩罚项，防止权重过大                      | $J(\\theta) = \\text{Loss} + \\lambda \\sum_{i} \\theta_i^2$         | 防止权重过大，提升模型稳定性，适用于多重共线性问题      |\n",
    "| **Dropout**            | 训练中随机丢弃部分神经元，减少神经元间的共适应性                     | -                                     | 强迫网络学习冗余特征，类似集成学习效果，但增加训练时间   |\n",
    "| **早停法（Early Stopping）** | 根据验证集性能停止训练，防止过度拟合训练数据                     | -                                     | 简单易行，需验证集监控，可能错过最佳模型                |\n",
    "| **数据增强（Data Augmentation）** | 对训练数据随机变换（如旋转/裁剪），扩展数据集多样性          | -                                     | 提升模型鲁棒性，尤其适用于图像/文本数据，但增加计算开销 |\n",
    "| **批归一化（Batch Normalization）** | 对每层输入归一化处理，稳定数据分布                        | -                                     | 加速训练并间接正则化，减少对初始权重的敏感度       |\n",
    "| **权重衰减（Weight Decay）** | 优化器中加入权重衰减项，等价于L2正则化                          | 同L2正则化                            | 直接约束权重幅度，与优化器（如AdamW）结合使用              |\n",
    "| **剪枝（Pruning）**    | 移除训练后不重要的连接或神经元，压缩模型规模                         | -                                     | 减少推理计算量，适合资源受限设备，但需额外训练步骤              |\n",
    "\n",
    "下面示例代码中使用了L2正则化机制\n",
    "\n",
    "**L2正则化机制**\n",
    "\n",
    "1. **L2正则化的定义**\n",
    "\n",
    "- L2正则化在损失函数中添加权重平方和的惩罚项，新的损失函数为：$L = \\text{MSE} + \\lambda \\left( \\|W_1\\|_2^2 + \\|W_2\\|_2^2 + \\|W_3\\|_2^2 \\right)$\n",
    "\n",
    "  其中：\n",
    "  - $\\text{MSE} = \\frac{1}{\\text{batch\\_size}} \\sum_{i=1}^{\\text{batch\\_size}} (y_{\\text{pred},i} - y_i)^2$ 是均方误差。\n",
    "  - $\\|W_k\\|_2^2 = \\sum_{i,j} W_{k,ij}^2$ 是权重矩阵的 L2 范数的平方。\n",
    "  - $\\lambda = \\text{l2\\_lambda}$ 是正则化系数，控制正则化强度。\n",
    "  - \n",
    "- 代码中添加了超参数l2_lambda = 0.01，表示正则化强度。\n",
    "\n",
    "2. **损失计算中的改动**\n",
    "   损失代码中添加了L2正则化项。\n",
    "   ```python\n",
    "   l2_loss = l2_lambda * (np.sum(W1 ** 2) + np.sum(W2 ** 2) + np.sum(W3 ** 2))\n",
    "   batch_loss = mse_loss + l2_loss\n",
    "   ```\n",
    "3. **梯度计算中的改动**\n",
    "   - L2 正则化对权重 $ W_k $ 的梯度贡献为：$$\\frac{\\partial}{\\partial W_k} \\left( \\lambda \\|W_k\\|_2^2 \\right) = 2 \\lambda W_k$$\n",
    "   - 因此，权重梯度需要加上 L2 正则化项：\n",
    "     - 原梯度：dW3 = a2.T @ dloss\n",
    "     - 新梯度：dW3 = a2.T @ dloss + 2 * l2_lambda * W3\n",
    "     - 类似地，dW2 和 dW1 也增加正则化项\n",
    "   - 偏置（b1, b2, b3）不参与 L2 正则化，梯度计算不变。\n",
    "\n",
    "4. **正则化的效果**\n",
    "   - L2 正则化通过惩罚大权重值，鼓励模型学习较小的权重，减少模型复杂度，降低过拟合风险。\n",
    "   - 代码中设置 l2_lambda = 0.01，这是一个适中的正则化强度，可以根据实验效果调整（例如 0.001 或 0.1）。\n",
    "\n",
    "\n",
    " \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df7db1bb-32d8-4c88-8921-3aa521b9a86d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# 设置随机种子，确保可重复性\n",
    "np.random.seed(42)\n",
    "\n",
    "# ---------- 模拟数据 ----------\n",
    "N, D_in, H1, H2, D_out = 256, 20, 50, 50, 1  # 样本数量256，输入维度20，隐藏层同为50，输出维度为1\n",
    "batch_size = 32  # 每个 mini-batch 的大小\n",
    "\n",
    "# 输入 X 和真实标签 y，形状分别为 (N, D_in) 和 (N, 1)\n",
    "X = np.random.randn(N, D_in)\n",
    "y = np.random.randn(N, D_out)\n",
    "\n",
    "# ---------- 权重初始化 ----------\n",
    "W1 = np.random.randn(D_in, H1) * 0.01\n",
    "b1 = np.zeros((1, H1))\n",
    "W2 = np.random.randn(H1, H2) * 0.01\n",
    "b2 = np.zeros((1, H2))\n",
    "W3 = np.random.randn(H2, D_out) * 0.01\n",
    "b3 = np.zeros((1, D_out))\n",
    "\n",
    "# ---------- 超参数 ----------\n",
    "learning_rate = 0.01\n",
    "epochs = 500\n",
    "num_batches = N // batch_size  # 计算 batch 数量\n",
    "l2_lambda = 0.01  # L2 正则化系数\n",
    "\n",
    "# ---------- 激活函数 ----------\n",
    "def relu(x):\n",
    "    return np.maximum(0, x)\n",
    "\n",
    "def relu_grad(x):\n",
    "    return (x > 0).astype(float)\n",
    "\n",
    "# ---------- 训练循环 ----------\n",
    "for epoch in range(epochs):\n",
    "    # 打乱数据（每个 epoch 随机化样本顺序）\n",
    "    indices = np.random.permutation(N)\n",
    "    X_shuffled = X[indices]\n",
    "    y_shuffled = y[indices]\n",
    "    \n",
    "    # 初始化 epoch 总损失\n",
    "    epoch_loss = 0.0\n",
    "    \n",
    "    # 按 mini-batch 迭代\n",
    "    for i in range(num_batches):\n",
    "        # 获取当前 batch 的数据\n",
    "        start_idx = i * batch_size\n",
    "        end_idx = start_idx + batch_size\n",
    "        X_batch = X_shuffled[start_idx:end_idx]  # (batch_size, D_in)\n",
    "        y_batch = y_shuffled[start_idx:end_idx]  # (batch_size, D_out)\n",
    "        \n",
    "        # ---- 前向传播 ----\n",
    "        z1 = X_batch @ W1 + b1      # (batch_size, H1)\n",
    "        a1 = relu(z1)               # (batch_size, H1)\n",
    "        z2 = a1 @ W2 + b2           # (batch_size, H2)\n",
    "        a2 = relu(z2)               # (batch_size, H2)\n",
    "        z3 = a2 @ W3 + b3           # (batch_size, D_out)\n",
    "        y_pred = z3                 # 输出层无激活函数（线性输出）\n",
    "        \n",
    "        # ---- 计算损失（包括 L2 正则化项） ----\n",
    "        mse_loss = np.mean((y_pred - y_batch) ** 2)\n",
    "        # L2 正则化项：lambda * (||W1||^2 + ||W2||^2 + ||W3||^2)\n",
    "        l2_loss = l2_lambda * (np.sum(W1 ** 2) + np.sum(W2 ** 2) + np.sum(W3 ** 2))\n",
    "        batch_loss = mse_loss + l2_loss\n",
    "        epoch_loss += batch_loss\n",
    "        \n",
    "        # ---- 反向传播 ----\n",
    "        # MSE 损失的导数\n",
    "        dloss = 2 * (y_pred - y_batch) / batch_size\n",
    "        \n",
    "        # 第三层梯度（加上 L2 正则化项）\n",
    "        dW3 = a2.T @ dloss + 2 * l2_lambda * W3  # (H2, D_out)\n",
    "        db3 = np.sum(dloss, axis=0, keepdims=True)  # (1, D_out)\n",
    "        \n",
    "        # 第二层反传\n",
    "        da2 = dloss @ W3.T          # (batch_size, H2)\n",
    "        dz2 = da2 * relu_grad(z2)   # (batch_size, H2)\n",
    "        \n",
    "        dW2 = a1.T @ dz2 + 2 * l2_lambda * W2  # (H1, H2)\n",
    "        db2 = np.sum(dz2, axis=0, keepdims=True)\n",
    "        \n",
    "        # 第一层反传\n",
    "        da1 = dz2 @ W2.T            # (batch_size, H1)\n",
    "        dz1 = da1 * relu_grad(z1)   # (batch_size, H1)\n",
    "        \n",
    "        dW1 = X_batch.T @ dz1 + 2 * l2_lambda * W1  # (D_in, H1)\n",
    "        db1 = np.sum(dz1, axis=0, keepdims=True)\n",
    "        \n",
    "        # ---- 参数更新 ----\n",
    "        W3 -= learning_rate * dW3\n",
    "        b3 -= learning_rate * db3\n",
    "        W2 -= learning_rate * dW2\n",
    "        b2 -= learning_rate * db2\n",
    "        W1 -= learning_rate * dW1\n",
    "        b1 -= learning_rate * db1\n",
    "    \n",
    "    # 计算平均 epoch 损失\n",
    "    epoch_loss /= num_batches\n",
    "    \n",
    "    # 每 50 个 epoch 输出损失\n",
    "    if epoch % 50 == 0:\n",
    "        print(f\"Epoch {epoch:3d} | Loss: {epoch_loss:.6f}\")\n",
    "\n",
    "# ---------- 最终损失（全数据集） ----------\n",
    "z1 = X @ W1 + b1\n",
    "a1 = relu(z1)\n",
    "z2 = a1 @ W2 + b2\n",
    "a2 = relu(z2)\n",
    "z3 = a2 @ W3 + b3\n",
    "y_pred = z3\n",
    "mse_loss = np.mean((y_pred - y) ** 2)\n",
    "l2_loss = l2_lambda * (np.sum(W1 ** 2) + np.sum(W2 ** 2) + np.sum(W3 ** 2))\n",
    "final_loss = mse_loss + l2_loss\n",
    "print(f\"Final Loss: {final_loss:.6f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56903ce0-e7b7-47ef-b56a-03649cc8ea62",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## 基于PyTorch的神经网络构建与训练"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96ae08ef-2f08-4d07-86b3-7e031e0614b6",
   "metadata": {},
   "source": [
    "### PyTorch的前向传播、反向传播与计算图\n",
    "- 计算函数y=ReLU(Wx+b)，并对其进行反向传播和求梯度\n",
    "\n",
    "PyTorch直接提供了神经网络训练中用到的前向传播、反向传播、激活函数、损失函数和参数优化（优化器）等组件中的多种主流实现，我们可以在代码中直接调用而无须再手动进行编码实现。\n",
    "\n",
    "| **组件**               | **功能**                                                                 | **核心类/函数示例**                                 | **底层实现依赖**               |\n",
    "| :--------------------- | :----------------------------------------------------------------------- | :-------------------------------------------------- | :----------------------------- |\n",
    "| **Tensor**             | 多维数组，支持 GPU 加速和自动微分                                           | `torch.tensor([[1, 2], [3, 4]], dtype=torch.float)` | ATen 张量库（C++）     |\n",
    "| **Autograd**           | 自动微分引擎，动态构建计算图并计算梯度                                       | `x.requires_grad=True; y.backward()`                | Autograd 引擎（C++）    |\n",
    "| **nn.Module**          | 神经网络模块基类，封装层、损失函数等                                         | `class CNN(nn.Module): def __init__(self): ...`     | pybind11 绑定（Python-C++ 接口） |\n",
    "| **Optimizer**          | 优化算法库（SGD、Adam 等），更新模型参数                                     | `torch.optim.Adam(model.parameters(), lr=0.001)`    | C++ 优化器后端             |\n",
    "| **DataLoader**         | 高效数据加载，支持批处理、乱序、并行加载                                      | `DataLoader(dataset, batch_size=64, shuffle=True)` | 多进程管理（C++）      |\n",
    "| **Distributed**        | 分布式训练支持（多 GPU/多节点）                                             | `DistributedDataParallel(model)`                    | gloo/NCCL 通信库（C++） |\n",
    "| **TorchScript**        | 模型编译为静态图，提升部署性能                                                | `scripted_model = torch.jit.script(model)`          | JIT 编译器（C++）      |\n",
    "| **扩展库**             | 领域专用工具库（CV/NLP/音频）                                               | `torchvision.models.resnet50()`                     | 独立 C++ 模块          |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "690ba14d-e8c0-4db3-915e-c4051d516d2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "# 定义输入、权重、偏置和目标值\n",
    "x = torch.tensor([[1.0, 2.0, 3.0]], requires_grad=True)  # 输入张量维度 (1, 3)\n",
    "W = torch.tensor([[0.5, -0.2, 0.1], [0.3, 0.4, -0.5]], requires_grad=True)  # 权重矩阵维度 (2, 3)\n",
    "b = torch.tensor([[0.1], [0.2]], requires_grad=True)  # 偏置向量 (2, 1)\n",
    "target = torch.tensor([[1.0], [0.5]])  # 目标值 (2, 1)\n",
    "\n",
    "# 前向传播\n",
    "# 示例函数：y=ReLU(Wx + b)\n",
    "def forward(x, W, b):\n",
    "    # 线性变换: z = Wx + b\n",
    "    z = torch.matmul(W, x.T) + b  # 矩阵乘法 + 偏置\n",
    "    # ReLU激活: y = ReLU(z)\n",
    "    y = torch.relu(z)\n",
    "    return y\n",
    "\n",
    "# 执行前向传播\n",
    "y = forward(x, W, b)\n",
    "print(\"Forward output (y):\", y)\n",
    "\n",
    "# 计算MSE损失\n",
    "loss = F.mse_loss(y, target)\n",
    "print(\"MSE Loss:\", loss.item())\n",
    "\n",
    "# 反向传播\n",
    "loss.backward()\n",
    "\n",
    "# 输出梯度\n",
    "print(\"Gradient of x:\", x.grad)\n",
    "print(\"Gradient of W:\", W.grad)\n",
    "print(\"Gradient of b:\", b.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "208527c0-74a1-4966-a24f-930b9d71906c",
   "metadata": {},
   "source": [
    "### 计算图的相关图形\n",
    "\n",
    "依赖于操作系统上安装好graphviz包，例如对于Ubuntu系统，可以使用如下命令进行。\n",
    "\n",
    "```bash\n",
    "sudo apt-get update\n",
    "sudo apt-get install graphviz\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52a50c16-4555-4d60-bf79-7d8a176821fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchviz import make_dot\n",
    "y = forward(x, W, b)\n",
    "loss = F.mse_loss(y, target)\n",
    "make_dot(loss).render(\"computational_graph\", format=\"png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e68032cf-7e57-451c-b246-439ede857631",
   "metadata": {},
   "source": [
    "### 基于PyTorch的神经网络构建与训练\n",
    "- 将前面示例“多层神经网络”修改为基于PyTorch的实现。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaa8ac48-e476-4d40-ae90-a27e627f4180",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "# 设置随机种子，确保可重复性\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# ---------- 模拟数据 ----------\n",
    "# 将 NumPy 数组转换为 PyTorch 张量\n",
    "X = torch.tensor([[1.0, 2.0], [3.0, 4.0]], dtype=torch.float32)  # shape: (2, 2)\n",
    "y = torch.tensor([[5.0], [11.0]], dtype=torch.float32)          # shape: (2, 1)\n",
    "\n",
    "N, D = X.shape\n",
    "\n",
    "# 网络结构参数\n",
    "H1 = 4  # 第 1 个隐藏层神经元个数\n",
    "H2 = 4  # 第 2 个隐藏层神经元个数\n",
    "lr = 0.01 # 学习率\n",
    "\n",
    "# ---------- 定义神经网络模型 ----------\n",
    "# PyTorch 的 nn.Module 是构建神经网络的基础\n",
    "class SimpleNN(nn.Module):\n",
    "    def __init__(self, input_size, H1_size, H2_size, output_size):\n",
    "        super(SimpleNN, self).__init__()\n",
    "        # 定义各层\n",
    "        # nn.Linear(in_features, out_features) 自动包含权重和偏置\n",
    "        self.fc1 = nn.Linear(input_size, H1_size)\n",
    "        self.relu1 = nn.ReLU() # ReLU 激活函数\n",
    "        self.fc2 = nn.Linear(H1_size, H2_size)\n",
    "        self.relu2 = nn.ReLU()\n",
    "        self.fc3 = nn.Linear(H2_size, output_size)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 定义前向传播路径\n",
    "        out = self.fc1(x)\n",
    "        out = self.relu1(out)\n",
    "        out = self.fc2(out)\n",
    "        out = self.relu2(out)\n",
    "        out = self.fc3(out)\n",
    "        return out\n",
    "\n",
    "# 实例化模型\n",
    "model = SimpleNN(D, H1, H2, 1)\n",
    "\n",
    "# ---------- 定义损失函数和优化器 ----------\n",
    "# 均方误差损失 (MSELoss)\n",
    "criterion = nn.MSELoss(reduction='mean') # reduction='mean'表示取批次的平均损失\n",
    "# Adam优化器，它会自动处理参数的梯度更新\n",
    "optimizer = optim.Adam(model.parameters(), lr=lr)\n",
    "\n",
    "# ---------- 单次训练迭代 ----------\n",
    "print(\"--- 进行单次训练迭代 ---\")\n",
    "\n",
    "# 前向传播\n",
    "y_hat = model(X)\n",
    "\n",
    "# 计算损失\n",
    "# PyTorch的MSELoss已经处理了0.5和求平均\n",
    "loss = criterion(y_hat, y)\n",
    "\n",
    "# 反向传播和优化\n",
    "optimizer.zero_grad() # 清除之前的梯度，避免累积\n",
    "loss.backward()       # 执行反向传播，计算所有可训练参数的梯度\n",
    "optimizer.step()      # 根据计算出的梯度更新模型参数\n",
    "\n",
    "print(f\"Loss: {loss.item():.4f}\")\n",
    "\n",
    "# ---------- 输出更新后的参数 (可选) ----------\n",
    "# 注意：PyTorch 权重的形状通常是 (out_features, in_features)，与 NumPy 中的 (in_features, out_features) 相反\n",
    "print(\"\\n--- 更新后的参数 ---\")\n",
    "print(\"W1 (fc1.weight):\\n\", model.fc1.weight.data)\n",
    "print(\"b1 (fc1.bias):\\n\", model.fc1.bias.data)\n",
    "print(\"\\nW2 (fc2.weight):\\n\", model.fc2.weight.data)\n",
    "print(\"b2 (fc2.bias):\\n\", model.fc2.bias.data)\n",
    "print(\"\\nW3 (fc3.weight):\\n\", model.fc3.weight.data)\n",
    "print(\"b3 (fc3.bias):\\n\", model.fc3.bias.data)\n",
    "\n",
    "\n",
    "# ---------- 验证最终预测 ----------\n",
    "with torch.no_grad(): # 在评估模式下，不计算梯度\n",
    "    y_pred_final = model(X)\n",
    "\n",
    "print(\"\\n--- 最终预测 ---\")\n",
    "print(\"原始数据 X:\\n\", X)\n",
    "print(\"原始标签 y:\\n\", y)\n",
    "print(\"预测值 y_hat:\\n\", y_pred_final)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7b7c23f-b311-4706-b12e-0a2a0f7dce2b",
   "metadata": {},
   "source": [
    "### 基于神经网络的多轮训练示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c6cc1ac-b0a4-457c-b066-9c08b710a554",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import numpy as np\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "# 设置随机种子，确保可重复性\n",
    "torch.manual_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "# ---------- 模拟数据 ----------\n",
    "N, D_in, H1, H2, D_out = 256, 20, 50, 50, 1  # 样本数量256，输入维度20，隐藏层同为50，输出维度为1\n",
    "batch_size = 32  # 每个 mini-batch 的大小\n",
    "\n",
    "# 输入X和真实标签y，转换为PyTorch张量\n",
    "X = torch.randn(N, D_in)\n",
    "y = torch.randn(N, D_out)\n",
    "\n",
    "# 创建PyTorch数据集和 DataLoader\n",
    "dataset = TensorDataset(X, y)\n",
    "dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)  # 随机打乱\n",
    "\n",
    "# ---------- 定义神经网络模型 ----------\n",
    "class NeuralNetwork(nn.Module):\n",
    "    def __init__(self, D_in, H1, H2, D_out):\n",
    "        super(NeuralNetwork, self).__init__()\n",
    "        # 第一隐藏层\n",
    "        self.linear1 = nn.Linear(D_in, H1)\n",
    "        self.relu1 = nn.ReLU()\n",
    "        # 第二隐藏层\n",
    "        self.linear2 = nn.Linear(H1, H2)\n",
    "        self.relu2 = nn.ReLU()\n",
    "        # 输出层\n",
    "        self.linear3 = nn.Linear(H2, D_out)\n",
    "        \n",
    "        # 使用He初始化权重\n",
    "        nn.init.kaiming_normal_(self.linear1.weight, mode='fan_in', nonlinearity='relu')\n",
    "        nn.init.zeros_(self.linear1.bias)\n",
    "        nn.init.kaiming_normal_(self.linear2.weight, mode='fan_in', nonlinearity='relu')\n",
    "        nn.init.zeros_(self.linear2.bias)\n",
    "        nn.init.kaiming_normal_(self.linear3.weight, mode='fan_in', nonlinearity='relu')\n",
    "        nn.init.zeros_(self.linear3.bias)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # 前向传播\n",
    "        x = self.linear1(x)\n",
    "        x = self.relu1(x)\n",
    "        x = self.linear2(x)\n",
    "        x = self.relu2(x)\n",
    "        x = self.linear3(x)\n",
    "        return x\n",
    "\n",
    "# ---------- 超参数 ----------\n",
    "learning_rate = 0.01\n",
    "epochs = 500\n",
    "l2_lambda = 0.01  # L2正则化系数\n",
    "\n",
    "# ---------- 初始化模型、损失函数和优化器 ----------\n",
    "model = NeuralNetwork(D_in, H1, H2, D_out)\n",
    "criterion = nn.MSELoss(reduction='mean')  # 均方误差损失\n",
    "# 使用SGD优化器，weight_decay实现L2正则化\n",
    "optimizer = optim.SGD(model.parameters(), lr=learning_rate, weight_decay=l2_lambda)\n",
    "\n",
    "# ---------- 训练循环 ----------\n",
    "model.train()  # 设置模型为训练模式\n",
    "for epoch in range(epochs):\n",
    "    epoch_loss = 0.0\n",
    "    num_batches = len(dataloader)\n",
    "    \n",
    "    # 遍历 mini-batches\n",
    "    for X_batch, y_batch in dataloader:\n",
    "        # 前向传播\n",
    "        y_pred = model(X_batch)\n",
    "        \n",
    "        # 计算损失（MSE损失，L2正则化由优化器自动处理）\n",
    "        mse_loss = criterion(y_pred, y_batch)\n",
    "        epoch_loss += mse_loss.item()\n",
    "        \n",
    "        # 反向传播和优化\n",
    "        optimizer.zero_grad()  # 清空梯度\n",
    "        mse_loss.backward()    # 计算梯度\n",
    "        optimizer.step()       # 更新参数\n",
    "    \n",
    "    # 计算平均 epoch 损失\n",
    "    epoch_loss /= num_batches\n",
    "    \n",
    "    # 每 50 个 epoch 输出损失\n",
    "    if epoch % 50 == 0:\n",
    "        print(f\"Epoch {epoch:3d} | Loss: {epoch_loss:.6f}\")\n",
    "\n",
    "# ---------- 最终损失（全数据集） ----------\n",
    "model.eval()  # 设置模型为推理模式\n",
    "with torch.no_grad():\n",
    "    y_pred = model(X)\n",
    "    mse_loss = criterion(y_pred, y)\n",
    "    # 计算 L2 正则化项\n",
    "    l2_loss = 0.0\n",
    "    for param in model.parameters():\n",
    "        if param.requires_grad and len(param.shape) > 1:  # 仅对权重矩阵应用L2正则化\n",
    "            l2_loss += torch.sum(param ** 2)\n",
    "    l2_loss *= l2_lambda\n",
    "    final_loss = mse_loss + l2_loss\n",
    "    print(f\"Final Loss: {final_loss.item():.6f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f7269f-68bd-4b46-9320-a74803a68efc",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## 张量及其自动求导\n",
    "\n",
    "在PyTorch的世界里，张量（Tensor） 是所有操作的基石，就像NumPy中的ndarray一样。然而，PyTorch的张量远不止一个多维数组那么简单，它还承载着在GPU上加速计算以及自动求导的核心能力，这些都是深度学习的命脉。\n",
    "\n",
    "简单来说，张量就是一个多维数组，可以存储各种类型的数据，从简单的标量（一个数字）到复杂的图像、视频数据，甚至神经网络的权重和梯度。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95add54e-0e4d-4d74-8b69-1dd5d7fbf652",
   "metadata": {},
   "source": [
    "### 张量创建\n",
    "\n",
    "张量支持多种创建方式，包括从已有数据创建、使用函数创建、从文件加载等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8c80f60-c8bb-459f-9ed2-f6dfc18ad966",
   "metadata": {},
   "source": [
    "#### 从 Python 列表或 NumPy 数组创建"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b608e9f3-e1d1-4cb7-9f7a-51aa20ece603",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "# 从Python列表创建\n",
    "data = [[1, 2], [3, 4]]\n",
    "x_data = torch.tensor(data)\n",
    "print(\"从列表创建:\\n\", x_data)\n",
    "\n",
    "# 从NumPy数组创建\n",
    "np_array = np.array(data)\n",
    "x_np = torch.from_numpy(np_array) # 注意：会共享内存\n",
    "print(\"从NumPy数组创建:\\n\", x_np)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcfbf5b4-7198-40f3-bc17-18581fe8edf9",
   "metadata": {},
   "source": [
    "#### 创建全0或全1张量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87fe7d89-0af5-4549-af74-fc8a245cec50",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2 行 3 列的全一张量\n",
    "x_ones = torch.ones(2, 3)\n",
    "print(\"全一张量:\\n\", x_ones)\n",
    "\n",
    "# 2 行 3 列的全零张量\n",
    "x_zeros = torch.zeros(2, 3)\n",
    "print(\"全零张量:\\n\", x_zeros)\n",
    "\n",
    "# 指定值\n",
    "x_full = torch.full((2, 3), 7)\n",
    "print(\"全指定值张量:\\n\", x_full)\n",
    "\n",
    "# 单位矩阵/对角矩阵\n",
    "x_eye = torch.eye(3)\n",
    "x_diag = torch.diag(torch.tensor([1, 2, 3]))\n",
    "print(\"单位矩阵:\\n\", x_eye)\n",
    "print(\"对角矩阵:\\n\", x_diag)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfa46358-54fb-4850-a0df-f87bc77fbb2e",
   "metadata": {},
   "source": [
    "#### 创建随机张量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88d093a5-8716-4f8e-9c41-34f24defd85c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2 行 3 列的随机张量 (0到1之间均匀分布)\n",
    "x_rand = torch.rand(2, 3)\n",
    "print(\"均匀分布随机张量:\\n\", x_rand)\n",
    "\n",
    "# 2行3列的随机张量 (标准正态分布)\n",
    "x_randn = torch.randn(2, 3)\n",
    "print(\"正态分布随机张量:\\n\", x_randn)\n",
    "\n",
    "# [0,10)之间的随机数构成的张量，维度2x3\n",
    "torch.randint(0, 10, (2, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e6019c8-844b-4750-a24e-d4b8f5f1161a",
   "metadata": {},
   "source": [
    "#### 特定分布初始化\n",
    "torch.normal()可用于生成符合正态分布（高斯分布）的随机数张量，它支持多种参数组合，可根据需求灵活生成不同均值、标准差和形状的随机数据。\n",
    "\n",
    "**核心参数**\n",
    "- mean：正态分布的均值（标量或张量）。\n",
    "- std：正态分布的标准差（标量或张量）。\n",
    "- size：输出张量的形状（元组），当 mean 和 std 均为标量时必须指定。\n",
    "- generator：控制随机数生成的种子（可选）。\n",
    "- out：指定输出张量（可选）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d7fd53e-056f-4d4d-92f0-b4d821fd122f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 生成形状为 (2, 3) 的标准正态分布随机数（均值为0，标准差为1）\n",
    "tensor = torch.normal(mean=0.0, std=1.0, size=(2, 3))\n",
    "print(\"准正态分布随机数（均值为0，标准差为1）的随机张量:\\n\", tensor)\n",
    "\n",
    "# 使用generator参数（随机数种子）确保结果可复现\n",
    "gen = torch.Generator().manual_seed(42)\n",
    "tensor = torch.normal(mean=0, std=1, size=(2,2), generator=gen)\n",
    "print(\"准正态分布随机数（均值为0，标准差为1）的随机张量，可复现:\\n\", tensor)\n",
    "\n",
    "# 使用示例1：神经网络权重初始化\n",
    "weights = torch.normal(mean=0.0, std=0.01, size=(100, 200))  # 高斯初始化\n",
    "\n",
    "# 使用示例2：数据增强（添加噪声）\n",
    "clean_data = torch.randn(10, 3, 224, 224)  # 原始数据\n",
    "noisy_data = clean_data + torch.normal(mean=0, std=0.1, size=clean_data.shape)   # .shape方法用于获取指定tensor的形状\n",
    "\n",
    "# 生成线性回归数据集：y = Xw + b + 噪声\n",
    "X = torch.normal(0, 1, (100, 2))\n",
    "y = X @ torch.tensor([3.0, -2.0]) + 1.5 + torch.normal(0, 0.1, (100,))\n",
    "print(\"生成的回归数据集:\\n\", y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f21d50d6-4c8b-4cd1-8571-7db1a2ba73fd",
   "metadata": {},
   "source": [
    "#### 使用 arange()/linspace()/logspace()\n",
    "- torch.arange()：创建等差数列（整数或浮点数）\n",
    "- torch.linspace()：创建均匀间隔的线性序列\n",
    "- torch.logspace()：创建对数尺度上的均匀序列\n",
    "\n",
    "**常用参数**\n",
    "- start：指数起始值\n",
    "- end：指数结束值（包含）\n",
    "- steps：生成的元素个数\n",
    "- base：对数底数（默认10），logspace()专用\n",
    "- dtype：数据类型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad97b2d7-e902-48e3-9523-8b2d8677179f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 浮点数序列：1.0到4.0（步长0.5）\n",
    "a = torch.arange(1, 4, 0.5)             # tensor([1.0, 1.5, 2.0, 2.5, 3.0, 3.5])\n",
    "# 指定设备和数据类型\n",
    "b = torch.arange(0, 6, 2, dtype=torch.float32, device='cpu')  # tensor([0., 2., 4.], device='cuda:0')\n",
    "\n",
    "# 生成0到1之间的5个等分点\n",
    "c = torch.linspace(0, 1, 5)             # tensor([0.0000, 0.2500, 0.5000, 0.7500, 1.0000])\n",
    "# 生成-1到1之间的3个点\n",
    "d = torch.linspace(-1, 1, 3)             # tensor([-1., 0., 1.])\n",
    "\n",
    "# 生成10^0到10^2之间的5个点（底数10）\n",
    "e = torch.logspace(0, 2, 5)             # tensor([1.0000, 3.1623, 10.0000, 31.6228, 100.0000])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba70c4b5-ca37-4522-9274-946e8690b0af",
   "metadata": {},
   "source": [
    "#### 克隆和复制\n",
    "创建的新张量与原张量数据相同，但不共享内存，修改副本不影响原张量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0cb6a63-7d22-408c-980e-0edbb361f3cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.tensor([1, 2, 3])\n",
    "b = a.clone()\n",
    "b[0] = 100\n",
    "print(a)  # tensor([1, 2, 3])  原张量不变\n",
    "print(b)  # tensor([100, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bebdd9b8-68a8-46fd-bfcb-b79ae735ba13",
   "metadata": {},
   "source": [
    "#### 从文件加载Tensor\n",
    "\n",
    "首先生成示例CSV文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fd354e5-8c59-4018-811b-9ea1fa82d078",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# 创建2D数值数组（例如3x3矩阵）\n",
    "data = np.array([\n",
    "    [1, 2, 3],\n",
    "    [4, 5, 6],\n",
    "    [7, 8, 9]\n",
    "])\n",
    "\n",
    "# 保存为CSV，控制格式和分隔符\n",
    "np.savetxt(\"example_csv_data.csv\", data, \n",
    "           delimiter=\",\",   # 逗号分隔\n",
    "           fmt=\"%d\",        # 整数格式（避免科学计数）\n",
    "           #header=\"Col1,Col2,Col3\",  # 自定义列名\n",
    "           comments=\"\")     # 避免header被注释"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "412c55d0-3ffc-4a44-af4d-fb539d6d453a",
   "metadata": {},
   "source": [
    "#### 从csv加载tensor示例\n",
    "\n",
    "需要首先由NumPy加载，而后再使用.from_numpy()方法转换为Tensor。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86f681e1-1987-44b0-a6a9-6c77468ef8b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 读取CSV（无标题行）\n",
    "array = np.loadtxt(\"example_csv_data.csv\", delimiter=\",\")\n",
    "# 所有字段必须为数值型，否则.float()方法可能会失败\n",
    "tensor = torch.from_numpy(array).float()\n",
    "\n",
    "print(tensor.shape)  # e.g., [3, 3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9794cd45-134d-4363-bb62-09f6142766be",
   "metadata": {},
   "source": [
    "#### 加载图像文件为Tensor\n",
    "\n",
    "适用于计算视觉类的任务，需要通过torchvision模型的read_image进行。\n",
    "\n",
    "首先，我们要先安装好torchvision模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f98a074b-3771-48db-9fae-dd2b21d52ea7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install torchvision torchaudio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7da2fc1-9ec0-40cf-8abe-dd20764c957e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.io import read_image\n",
    "\n",
    "img_tensor = read_image(\"computational_graph.png\")  # 结果为[C, H, W]格式，uint8类型\n",
    "img_tensor = img_tensor.float() / 255.0  # 转为float并归一化\n",
    "\n",
    "print(img_tensor.shape)  # e.g., [3, 224, 224]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85e8a389-08d9-4a57-b135-67146ba2e224",
   "metadata": {},
   "source": [
    "#### 加载音频文件为Tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cffbbfc0-facb-470f-abc4-91af2c5031d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchaudio\n",
    "\n",
    "waveform, sample_rate = torchaudio.load(\"audio.wav\")  # waveform是 [channels, time]\n",
    "\n",
    "print(waveform.shape)      # e.g., [1, 16000]\n",
    "print(sample_rate)         # 采样率"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "813faef4-b760-464d-a8a1-8e53b1678702",
   "metadata": {},
   "source": [
    "### 张量支持的操作"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83317755-1304-4c12-8822-deb0279a5bca",
   "metadata": {},
   "source": [
    "#### 索引与切片"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5ce002d-37ef-4326-8212-17d24bf81a91",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([[1, 2, 3], [4, 5, 6]])\n",
    "\n",
    "print(x[0])        # 第一行\n",
    "print(x[:, 1])     # 第二列\n",
    "print(x[1, 2])     # 第2行第3列的元素\n",
    "print(x[:, 1:])    # 所有行，第2列到最后\n",
    "\n",
    "# 高级索引\n",
    "indices = torch.tensor([0, 2])\n",
    "print(x[0, indices])  # 第1行中的第1列和第3列"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "577f7e59-7b37-4db5-a7b8-fbcd79f672d4",
   "metadata": {},
   "source": [
    "#### 变形操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5a9c085-401e-4e17-880f-e114426442d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.arange(12)\n",
    "\n",
    "print(\"原始:\", x)\n",
    "print(\"reshape:\", x.view(3, 4))\n",
    "print(\"添加维度:\", x.unsqueeze(0).shape)\n",
    "print(\"去除维度:\", x.unsqueeze(0).squeeze().shape)\n",
    "print(\"转置:\", torch.randn(2, 3).t())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ae53535-2bf3-40e8-b92b-12271e7380ad",
   "metadata": {},
   "source": [
    "#### 连接操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72f89030-a69b-4c30-9ad7-9186c6771a35",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.ones(2, 3)\n",
    "b = torch.zeros(2, 3)\n",
    "\n",
    "print(\"拼接 (dim=0):\", torch.cat([a, b], dim=0))  # 4x3\n",
    "print(\"拼接 (dim=1):\", torch.cat([a, b], dim=1))  # 2x6\n",
    "\n",
    "# 堆叠（增加新维度）\n",
    "print(\"堆叠:\", torch.stack([a, b], dim=0))  # 2x2x3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "740f4f8e-eea4-47ae-bfbb-73437e02ffe8",
   "metadata": {},
   "source": [
    "#### 数学操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3e54369-3985-4395-aa9c-86bed8001da8",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([1.0, 2.0, 3.0])\n",
    "y = torch.tensor([4.0, 5.0, 6.0])\n",
    "\n",
    "print(\"加:\", x + y)\n",
    "print(\"乘:\", x * y)\n",
    "print(\"点积:\", torch.dot(x, y))\n",
    "print(\"矩阵乘:\", torch.matmul(x.unsqueeze(0), y.unsqueeze(1)))  # 1x1\n",
    "print(\"指数:\", torch.exp(x))\n",
    "print(\"对数:\", torch.log(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae83726d-6451-4def-82f0-8eb8eada342c",
   "metadata": {},
   "source": [
    "#### 聚合操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12e72b64-0f07-450c-b9ee-7b1a54c006d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor([[1, 2], [3, 4]])\n",
    "\n",
    "print(\"总和:\", x.sum())\n",
    "print(\"均值:\", x.float().mean())\n",
    "print(\"最小/最大:\", x.min(), x.max())\n",
    "print(\"按行求和:\", x.sum(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a255902-1b97-4515-bc42-206008c5d27a",
   "metadata": {},
   "source": [
    "### 自动求导机制"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8de88195-a820-42d2-a9c1-8011e302877e",
   "metadata": {},
   "source": [
    "#### 自动求导机制示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8181b332-1b30-4bf6-b8ae-77c1064e73c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "## 第一部分：NumPy与PyTorch Tensor\n",
    "\n",
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "# 从 NumPy 创建 Tensor\n",
    "np_array = np.array([[1.0, 2.0], [3.0, 4.0]])\n",
    "tensor_from_np = torch.from_numpy(np_array)\n",
    "print(\"NumPy转Tensor：\\n\", tensor_from_np)\n",
    "\n",
    "# 从 Tensor 创建 NumPy\n",
    "tensor = torch.tensor([[5.0, 6.0], [7.0, 8.0]])\n",
    "np_from_tensor = tensor.numpy()\n",
    "print(\"Tensor转NumPy：\\n\", np_from_tensor)\n",
    "\n",
    "# 修改会同步（共享内存）\n",
    "tensor_from_np[0, 0] = 99\n",
    "print(\"修改后的NumPy：\\n\", np_array)\n",
    "\n",
    "## 第二部分：什么是自动求导机制？\n",
    "\n",
    "# NumPy不支持自动求导，只能手动写导数\n",
    "def f_np(x):\n",
    "    return x**2 + 3*x + 5\n",
    "\n",
    "x_val = 2.0\n",
    "delta = 1e-5\n",
    "numerical_grad = (f_np(x_val + delta) - f_np(x_val - delta)) / (2 * delta)\n",
    "print(\"数值求导结果（NumPy手动）：\", numerical_grad)\n",
    "\n",
    "## 第三部分：PyTorch自动求导机制\n",
    "\n",
    "# PyTorch 中的自动求导机制\n",
    "x = torch.tensor([2.0], requires_grad=True)\n",
    "y = x**2 + 3 * x + 5\n",
    "y.backward()  # 自动反向传播\n",
    "print(\"y对x的梯度值：\", x.grad)\n",
    "\n",
    "# 自动求导支持更复杂的模型\n",
    "x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)\n",
    "w = torch.tensor([0.1, 0.2, 0.3], requires_grad=True)\n",
    "b = torch.tensor([0.5], requires_grad=True)\n",
    "\n",
    "# 线性模型\n",
    "y_pred = x @ w + b\n",
    "loss = y_pred.sum()\n",
    "loss.backward()\n",
    "\n",
    "print(\"w的梯度：\", w.grad)\n",
    "print(\"b的梯度：\", b.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7494059-09de-4b62-a2dd-a10623d7c1fa",
   "metadata": {},
   "source": [
    "#### Tensor上的grad_fn属性\n",
    "\n",
    "- 每个具有requires_grad=True的张量（除了用户自己创建的叶子张量外）都有一个**grad_fn**属性，该属性是非叶子张量专有\n",
    "- 该属性指向一个函数，这个函数记录了创建当前张量所用的操作，并且知道如何计算该操作的梯度\n",
    "\n",
    "\n",
    "\n",
    "以下脚本执行后返回的MulBackward0和AddBackward0就是PyTorch内部用来计算梯度链式法则的函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1559cea8-1720-4042-88dd-45ebdcae1acc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "x = torch.tensor([1.0], requires_grad=True) # 叶子张量\n",
    "print(f\"x.grad_fn: {x.grad_fn}\")\n",
    "\n",
    "y = x * 2 # 非叶子张量，由乘法操作生成\n",
    "print(f\"y.grad_fn: {y.grad_fn}\")\n",
    "\n",
    "z = y + 3 # 非叶子张量，由加法操作生成\n",
    "print(f\"z.grad_fn: {z.grad_fn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "106deb1e-69d7-44bf-b95b-7625479fa836",
   "metadata": {},
   "source": [
    "#### grad属性\n",
    "\n",
    "loss.backward()被调用后，requires_grad=True的**叶子张量**的.grad属性会累积其梯度值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "242cff2f-49b6-4de5-bf28-7af485a2f318",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "x = torch.tensor([2.0], requires_grad=True)\n",
    "y = x**2 + 5 * x + 3\n",
    "y.backward() # 计算梯度\n",
    "\n",
    "print(f\"x 的梯度: {x.grad}\") # dy/dx = 2x + 5 = 2*2 + 5 = 9"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "896a552c-6e77-4581-a5ba-f84ebbc48036",
   "metadata": {},
   "source": [
    "#### 自动求导的过程示例\n",
    "\n",
    "为了详细说明PyTorch的自动求导机制，以函数$ y = W(\\text{ReLU}(WX + b)) + b $为例，我们将分析哪些是叶张量（leaf tensors）、哪些是非叶张量（non-leaf tensors），以及自动求导的完整过程。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd69ceda-4fbb-4d42-9e75-1ff8e5731ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "# 定义张量\n",
    "X = torch.randn(2, 3, requires_grad=True)    # 输入张量，形状 (2, 3)\n",
    "W1 = torch.randn(4, 3, requires_grad=True)   # 第一层权重，形状 (4, 3)\n",
    "b1 = torch.randn(4, requires_grad=True)      # 第一层偏置，形状 (4,)\n",
    "W2 = torch.randn(4, 4, requires_grad=True)   # 第二层权重，形状 (4, 4)\n",
    "b2 = torch.randn(4, requires_grad=True)      # 第二层偏置，形状 (4,)\n",
    "\n",
    "# 前向计算\n",
    "z = torch.matmul(X, W1.t()) + b1             # W1X + b1，形状 (2, 4)\n",
    "\n",
    "z_relu = torch.relu(z)                       # ReLU(W1X + b1)，形状 (2, 4)\n",
    "\n",
    "y = torch.matmul(z_relu, W2) + b2            # W2(ReLU(W1X + b1)) + b2，形状 (2, 4)\n",
    "\n",
    "loss = y.sum()                               # 标量损失\n",
    "\n",
    "# 反向传播\n",
    "loss.backward()\n",
    "\n",
    "# 打印叶张量信息\n",
    "print(\"X.is_leaf:\", X.is_leaf)               # True\n",
    "print(\"W1.is_leaf:\", W1.is_leaf)             # True\n",
    "print(\"b1.is_leaf:\", b1.is_leaf)             # True\n",
    "print(\"W2.is_leaf:\", W2.is_leaf)             # True\n",
    "print(\"b2.is_leaf:\", b2.is_leaf)             # True\n",
    "print(\"z.is_leaf:\", z.is_leaf)               # False\n",
    "print(\"z_relu.is_leaf:\", z_relu.is_leaf)     # False\n",
    "print(\"y.is_leaf:\", y.is_leaf)               # False\n",
    "print(\"loss.is_leaf:\", loss.is_leaf)         # False\n",
    "\n",
    "# 打印非叶张量的grad_fn\n",
    "print(f\"非叶张量z的grad_fn:\\n{z.grad_fn}\")\n",
    "print(f\"非叶张量z_relu的grad_fn:\\n{z_relu.grad_fn}\")\n",
    "print(f\"非叶张量y的grad_fn:\\n{y.grad_fn}\")\n",
    "\n",
    "# 打印叶张量的grad\n",
    "print(\"X.grad:\", X.grad)                     # 梯度\n",
    "print(\"W1.grad:\", W1.grad)\n",
    "print(\"b1.grad:\", b1.grad)\n",
    "print(\"W2.grad:\", W2.grad)\n",
    "print(\"b2.grad:\", b2.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05348ce0-2dc0-4d73-b42c-3659449964bb",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## 构建并训练神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d483e79-35de-4f3f-a847-9423c1ad1d7e",
   "metadata": {},
   "source": [
    "### 构建和训练神经网络的基本步骤\n",
    "\n",
    "**构建神经网络的基本步骤**\n",
    "- 准备数据：使用torch.utils.data加载和预处理数据\n",
    "- 定义网络结构：继承 nn.Module 类，定义网络的层和前向传播逻辑\n",
    "- 初始化网络：设置层的参数（如输入/输出维度）和激活函数\n",
    "- 选择损失函数和优化器：如交叉熵损失和SGD 优化器\n",
    "- 训练网络：通过前向传播、计算损失、反向传播和参数更新完成训练\n",
    "- 评估模型：在测试集上验证模型性能\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "089efda6-d079-425a-b3f6-b8c9e46be771",
   "metadata": {},
   "source": [
    "#### 第一步：定义神经网络结构\n",
    "- 目标：设计神经网络的架构，指定层、激活函数和其他组件。\n",
    "- 主要模块：\n",
    "  - torch.nn.Module：所有神经网络的基类，用于定义自定义网络。\n",
    "  - torch.nn：提供构建块，如全连接层 (nn.Linear)、卷积层 (nn.Conv2d)、激活函数 (nn.ReLU)、池化层 (nn.MaxPool2d) 等。\n",
    "  - torch.nn.Sequential：用于快速构建按顺序连接的层。\n",
    "- 步骤：\n",
    "  - 继承 nn.Module，在 __init__ 中定义层，在 forward 方法中实现前向传播逻辑。\n",
    "  - 将层注册为类属性以自动跟踪参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cda3a19-85e0-43ea-87c9-da81d1885c50",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class SimpleNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SimpleNet, self).__init__()\n",
    "        self.fc1 = nn.Linear(784, 128)  # 输入 784，输出 128\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(128, 10)   # 输出 10 个类别\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.view(-1, 784)  # 展平输入\n",
    "        x = self.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e22f2142-ddd5-4ddc-9f3b-8826b4433f42",
   "metadata": {},
   "source": [
    "#### 第二步：准备数据\n",
    "- 目标：加载和预处理数据集，组织为适合训练的格式。\n",
    "- 主要模块：\n",
    "  - torch.utils.data.Dataset：定义数据集的接口，用于加载数据。\n",
    "  - torch.utils.data.DataLoader：批量加载数据，支持打乱和多线程。\n",
    "  - torchvision.datasets：提供常用数据集（如 MNIST、CIFAR-10）。\n",
    "  - torchvision.transforms：提供数据预处理和增强工具。\n",
    "- 步骤：\n",
    "  - 使用 Dataset 或现成数据集加载数据。\n",
    "  - 应用变换（如归一化、数据增强）。\n",
    "  - 使用DataLoader创建批量迭代器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2b1d4e3-7069-404f-bacd-b4f419618e91",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "transform = transforms.ToTensor()\n",
    "trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa5bd6dd-e5fa-4ff5-9d9b-d218c30d0000",
   "metadata": {},
   "source": [
    "#### 第三步：初始化模型、损失函数和优化器\n",
    "- 目标：设置模型实例、损失函数和优化算法。\n",
    "- 主要模块：\n",
    "  - torch.nn：提供损失函数，如 nn.CrossEntropyLoss（分类）、nn.MSELoss（回归）。\n",
    "  - torch.optim：提供优化器，如 optim.SGD、optim.Adam。\n",
    "- 步骤：\n",
    "  - 实例化网络并移动到设备（CPU/GPU）。\n",
    "  - 选择适合任务的损失函数。\n",
    "  - 配置优化器，传入模型参数和超参数（如学习率）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10633b86-2553-4837-9ba8-ef41ef776886",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "model = SimpleNet().to(device)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "103b9d01-5f36-4ed3-b014-c281cf1248ea",
   "metadata": {},
   "source": [
    "#### 第四步：训练模型\n",
    "- 目标：通过前向传播、损失计算、反向传播和参数更新训练模型。\n",
    "- 主要模块：\n",
    "  - torch.nn.Module：通过 model(input) 执行前向传播。\n",
    "  - torch.autograd：自动计算梯度（隐式使用）。\n",
    "  - torch.optim：优化器更新参数。\n",
    "- 步骤：\n",
    "  - 遍历数据集，执行以下循环：\n",
    "    1. 清零梯度（optimizer.zero_grad()）。\n",
    "    2. 前向传播，计算输出。\n",
    "    3. 计算损失（criterion(output, target)）。\n",
    "    4. 反向传播（loss.backward()）。\n",
    "    5. 更新参数（optimizer.step()）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6a9cfc6-bb08-4642-bd80-c38281dad300",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.train()\n",
    "for epoch in range(5):\n",
    "    running_loss = 0.0\n",
    "    for images, labels in trainloader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "        optimizer.zero_grad()\n",
    "        outputs = model(images)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        running_loss += loss.item()\n",
    "    print(f'Epoch {epoch+1}, Loss: {running_loss/len(trainloader):.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7b5187b-9dc0-4074-9bef-696b96c916f3",
   "metadata": {},
   "source": [
    "#### 第五步：评估模型\n",
    "- 目标：在测试集上验证模型性能。\n",
    "- 主要模块：\n",
    "  - torch.nn.Module：使用 model.eval() 切换到评估模式。\n",
    "  - torch.no_grad：禁用梯度计算，节省内存。\n",
    "- 步骤：\n",
    "  - 加载测试数据。\n",
    "  - 设置模型为评估模式（model.eval()）。\n",
    "  - 使用 torch.no_grad() 计算预测并评估指标（如准确率）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9023a61f-b33e-42f3-b025-a9b50248f1c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "testset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=False)\n",
    "\n",
    "model.eval()\n",
    "correct = 0\n",
    "total = 0\n",
    "with torch.no_grad():\n",
    "    for images, labels in testloader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "        outputs = model(images)\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted == labels).sum().item()\n",
    "print(f'Test Accuracy: {100 * correct / total:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "114de64b-9a0c-4b57-a869-cd32dd187147",
   "metadata": {},
   "source": [
    "#### 第六步：保存模型\n",
    "- 目标：保存训练好的模型参数或整个模型，并在需要时加载。\n",
    "- 主要模块：\n",
    "  - torch.nn.Module：提供 state_dict() 和 load_state_dict() 方法。\n",
    "  - torch.save 和 torch.load：用于序列化和反序列化。\n",
    "- 步骤：\n",
    "  - 保存模型参数（推荐）或整个模型。\n",
    "  - 加载参数并恢复模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "255a987d-db47-40fa-ae89-45f1c03c51b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 保存模型\n",
    "torch.save(model.state_dict(), 'model.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40601096-70c2-4a09-8cbe-a79c31a6ea1f",
   "metadata": {},
   "source": [
    "#### 加载模型的简单示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d47d9b50-6179-4bb4-9df2-af8e8ef364d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 加载模型\n",
    "model = SimpleNet().to(device)\n",
    "model.load_state_dict(torch.load('model.pth'))\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "008c4331-a11e-45b4-bfc2-b18efb22e881",
   "metadata": {},
   "source": [
    "### 完整的示例\n",
    "不包含模型的保存和加载。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ba3e8dd-2623-4c25-a686-03af6922e164",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "# 定义神经网络\n",
    "class SimpleNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SimpleNet, self).__init__()\n",
    "        self.fc1 = nn.Linear(28 * 28, 128)  # 输入层到隐藏层（784 -> 128）\n",
    "        self.relu = nn.ReLU()               # 激活函数\n",
    "        self.fc2 = nn.Linear(128, 64)       # 隐藏层（128 -> 64）\n",
    "        self.fc3 = nn.Linear(64, 10)        # 输出层（64 -> 10，10个类别）\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.view(-1, 28 * 28)  # 展平输入（batch_size, 1, 28, 28）-> (batch_size, 784)\n",
    "        x = self.relu(self.fc1(x))\n",
    "        x = self.relu(self.fc2(x))\n",
    "        x = self.fc3(x)          # 输出 logits\n",
    "        return x\n",
    "\n",
    "# 准备数据\n",
    "transform = transforms.ToTensor()\n",
    "trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)\n",
    "\n",
    "# 初始化网络、损失函数和优化器\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "model = SimpleNet().to(device)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)\n",
    "\n",
    "# 训练网络\n",
    "for epoch in range(5):  # 训练 5 个 epoch\n",
    "    running_loss = 0.0\n",
    "    for images, labels in trainloader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "        \n",
    "        optimizer.zero_grad()          # 清零梯度\n",
    "        outputs = model(images)        # 前向传播\n",
    "        loss = criterion(outputs, labels)  # 计算损失\n",
    "        loss.backward()               # 反向传播\n",
    "        optimizer.step()              # 更新参数\n",
    "        \n",
    "        running_loss += loss.item()\n",
    "    print(f'Epoch {epoch+1}, Loss: {running_loss/len(trainloader):.4f}')\n",
    "\n",
    "print('训练完成！')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93c06dc0-8193-4e79-9dcd-119519c2a801",
   "metadata": {},
   "source": [
    "### PyTorch模型组合示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "357e320e-ef8d-4fbd-bd3a-ff88e624a268",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fe0443c-e2eb-4cd7-9369-9e76802a1d62",
   "metadata": {},
   "source": [
    "#### 使用nn.Sequential构建顺序网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b9b07c1-4ed3-45e7-8f4f-9d16df5d5ea1",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SequentialModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.model = nn.Sequential(\n",
    "            nn.Linear(784, 256),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(256, 10)\n",
    "        )\n",
    "    \n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "\n",
    "# 测试\n",
    "x = torch.randn(4, 784)\n",
    "seq_model = SequentialModel()\n",
    "print(\"Sequential output shape:\", seq_model(x).shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cdbcde4-fc8d-4dd2-94c9-a81279feae71",
   "metadata": {},
   "source": [
    "#### 使用 nn.ModuleList 构建重复结构网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a06b2e9-eec7-40fa-8695-3440b1c298e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModuleListModel(nn.Module):\n",
    "    def __init__(self, num_layers=3):\n",
    "        super().__init__()\n",
    "        self.input_layer = nn.Linear(784, 256)\n",
    "        self.hidden_layers = nn.ModuleList([\n",
    "            nn.Linear(256, 256) for _ in range(num_layers)\n",
    "        ])\n",
    "        self.output_layer = nn.Linear(256, 10)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.input_layer(x))\n",
    "        for layer in self.hidden_layers:\n",
    "            x = F.relu(layer(x))\n",
    "        return self.output_layer(x)\n",
    "\n",
    "# 测试\n",
    "x = torch.randn(4, 784)\n",
    "list_model = ModuleListModel()\n",
    "print(\"ModuleList output shape:\", list_model(x).shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d58a4b0a-0178-4769-a899-8cab5f295bf5",
   "metadata": {},
   "source": [
    "#### 使用 nn.ModuleDict 构建分支结构网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "992229f8-cd20-4244-b46a-de4ec412ef03",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModuleDictModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.shared = nn.Linear(784, 256)\n",
    "        self.heads = nn.ModuleDict({\n",
    "            'classification': nn.Linear(256, 10),\n",
    "            'regression': nn.Linear(256, 1)\n",
    "        })\n",
    "    \n",
    "    def forward(self, x, mode='classification'):\n",
    "        x = F.relu(self.shared(x))\n",
    "        return self.heads[mode](x)\n",
    "\n",
    "# 测试\n",
    "x = torch.randn(4, 784)\n",
    "dict_model = ModuleDictModel()\n",
    "print(\"ModuleDict classification head:\", dict_model(x, mode='classification').shape)\n",
    "print(\"ModuleDict regression head:\", dict_model(x, mode='regression').shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1dec1a5-993e-4e56-9f3d-149f37354ecd",
   "metadata": {},
   "source": [
    "### 激活函数示例\n",
    "\n",
    "**主流激活函数的特性对比**\n",
    "\n",
    "| 特性/函数        | Softmax                                    | Sigmoid                                  | Tanh                                     | ReLU                                   | GELU                                           | Swish                                          |\n",
    "| :--------------- | :----------------------------------------- | :--------------------------------------- | :--------------------------------------- | :------------------------------------- | :--------------------------------------------- | :--------------------------------------------- |\n",
    "| **数学表达式** | $P_i = \\frac{e^{z_i}}{\\sum_{j=1}^{K} e^{z_j}}$ | $\\sigma(x) = \\frac{1}{1 + e^{-x}}$       | $\\tanh(x) = \\frac{e^x - e^{-x}}{e^x + e^{-x}}$ | $f(x) = \\max(0, x)$                    | $f(x) = x \\cdot P(X \\le x)$ (标准正态CDF)       | $f(x) = x \\cdot \\sigma(\\beta x)$               |\n",
    "| **输出范围** | $(0, 1)$ (所有输出和为 1)                  | $(0, 1)$                                 | $(-1, 1)$                                | $[0, +\\infty)$                         | $(-\\infty, +\\infty)$                           | $(-\\infty, +\\infty)$                           |\n",
    "| **零中心输出** | 否（单独看每个输出）                       | 否                                       | **是** | 否                                     | **是** (输出均值趋近于 0)                  | **是** (输出均值趋近于 0)                  |\n",
    "| **平滑性** | 平滑                                       | 平滑                                     | 平滑                                     | 不平滑 (在 $x=0$ 处尖锐)             | **平滑** | **平滑** |\n",
    "| **单调性** | 无法直接判断 (是整个向量的转换)              | **单调递增** | **单调递增** | **单调递增** | **非单调** | **非单调** |\n",
    "| **是否存在梯度消失** | 否（通常不饱和），但有数值稳定性问题       | **是** (两端饱和区)                      | **是** (两端饱和区)                      | **否** (正值区)，但有“死亡 ReLU” | **否** (正值区，平滑过渡)，但计算复杂 | **否** (正负值区都有梯度)，但计算复杂 |\n",
    "| **“死亡神经元”问题** | 不适用                                     | 不适用                                   | 不适用                                   | **是** (负值区梯度为 0)                | 否                                             | 否                                             |\n",
    "| **计算成本** | 高 (涉及指数和求和)                        | 高 (涉及指数)                            | 高 (涉及指数)                            | **最低** (比较操作)                    | 较高 (涉及 $\\tanh$ 和多项式)                  | 较高 (涉及 Sigmoid 和乘法)                     |\n",
    "| **典型应用场景** | **多分类问题**的**输出层**，输出类别概率   | **二分类问题**的**输出层**，多标签分类 | 早期神经网络的**隐藏层**，RNNs             | **大多数隐藏层**，CNNs                 | **Transformer 架构** (BERT, GPT), 复杂模型  | **深层网络**，视觉任务 (图像分类、目标检测)    |\n",
    "| **主要优点** | 将输出转换为概率分布，适用于互斥多分类       | 输出概率化，可微                           | 输出零中心，收敛快于 Sigmoid           | 缓解梯度消失，计算高效，引入稀疏性     | 性能优越，平滑，自适应门控，缓解梯度消失       | 性能优越，平滑，非单调，自门控，无死亡 ReLU |\n",
    "| **主要缺点** | 数值稳定性，可能导致梯度接近 0               | 梯度消失，计算成本高，非零中心输出       | 梯度消失，计算成本高                     | “死亡 ReLU”，非零中心输出              | 计算成本高，概念复杂                           | 计算成本高                                     |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79b6da11-6244-4203-9253-892759a9c663",
   "metadata": {},
   "source": [
    "#### 示例1：使用不带激活函数的模型拟合纯性数据点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dc3e158-df4a-40f3-9fa7-2b492a6bc822",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import animation\n",
    "import numpy as np\n",
    "\n",
    "# ========== Step 1: 构造数据 ==========\n",
    "np.random.seed(0)\n",
    "x_np = np.linspace(-3, 3, 200)\n",
    "noise = np.random.normal(0, 0.3, x_np.shape)\n",
    "y_np = 2 * x_np + 1 + noise\n",
    "\n",
    "X_tensor = torch.tensor(x_np, dtype=torch.float32).view(-1, 1)\n",
    "Y_tensor = torch.tensor(y_np, dtype=torch.float32).view(-1, 1)\n",
    "\n",
    "# ========== Step 2: 定义模型 ==========\n",
    "class LinearRegressionModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(1, 1)  # y = wx + b\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.linear(x)\n",
    "\n",
    "model = LinearRegressionModel()\n",
    "\n",
    "# ========== Step 3: 定义损失函数和优化器 ==========\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.05)\n",
    "\n",
    "# ========== Step 4: 训练并记录每隔几步的预测 ==========\n",
    "epochs = 200\n",
    "history = []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    model.train()\n",
    "    optimizer.zero_grad()\n",
    "    output = model(X_tensor)\n",
    "    loss = criterion(output, Y_tensor)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 每 5 轮记录一次当前预测值\n",
    "    if epoch % 5 == 0 or epoch == epochs - 1:\n",
    "        with torch.no_grad():\n",
    "            pred_y = model(X_tensor).detach()\n",
    "            history.append(pred_y.numpy())\n",
    "\n",
    "print(\"训练完成，准备动画...\")\n",
    "\n",
    "# ========== Step 5: 动态可视化拟合过程 ==========\n",
    "fig, ax = plt.subplots(figsize=(8, 5))\n",
    "ax.scatter(x_np, y_np, c='blue', label=\"Data\")\n",
    "line, = ax.plot([], [], 'r-', linewidth=2, label=\"Prediction\")\n",
    "text = ax.text(0.05, 0.95, '', transform=ax.transAxes, fontsize=12, verticalalignment='top')\n",
    "ax.set_xlim(-3, 3)\n",
    "ax.set_ylim(min(y_np)-1, max(y_np)+1)\n",
    "ax.legend()\n",
    "ax.set_title(\"Linear Regression Fitting Animation\")\n",
    "\n",
    "def init():\n",
    "    line.set_data([], [])\n",
    "    text.set_text('')\n",
    "    return line, text\n",
    "\n",
    "def animate(i):\n",
    "    y_pred = history[i].flatten()\n",
    "    line.set_data(x_np, y_pred)\n",
    "    loss_val = np.mean((y_np - y_pred)**2)\n",
    "    text.set_text(f\"Epoch: {i*5:>3}, Loss: {loss_val:.4f}\")\n",
    "    return line, text\n",
    "\n",
    "ani = animation.FuncAnimation(fig, animate, init_func=init,\n",
    "                              frames=len(history), interval=150, blit=True)\n",
    "\n",
    "plt.close()  # 防止重复显示\n",
    "ani\n",
    "\n",
    "# 显示动画（在Jupyter中）\n",
    "HTML(ani.to_jshtml())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6993267-72ec-4b3e-b82e-de9df7982114",
   "metadata": {},
   "source": [
    "#### 示例2：使用不带激活函数的模型拟合非线性数据点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca2bfebf-2142-4d83-9fd2-7a1880f4880f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import animation\n",
    "import numpy as np\n",
    "\n",
    "# ========== Step 1: 生成非线性数据 ==========\n",
    "np.random.seed(0)\n",
    "x_np = np.linspace(-3, 3, 200)\n",
    "noise = np.random.normal(0, 0.5, x_np.shape)\n",
    "#y_np = x_np**2 + noise  # 非线性关系示例1：幂函数\n",
    "y_np = np.sin(x_np * 2) + noise  # 非线性关系示例2：三角函数\n",
    "\n",
    "X_tensor = torch.tensor(x_np, dtype=torch.float32).view(-1, 1)\n",
    "Y_tensor = torch.tensor(y_np, dtype=torch.float32).view(-1, 1)\n",
    "\n",
    "# ========== Step 2: 定义线性模型 ==========\n",
    "class LinearModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(1, 1)  # 线性模型 y = wx + b\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.linear(x)\n",
    "\n",
    "model = LinearModel()\n",
    "\n",
    "# ========== Step 3: 损失函数与优化器 ==========\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.05)\n",
    "\n",
    "# ========== Step 4: 训练并记录过程 ==========\n",
    "epochs = 200\n",
    "history = []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    model.train()\n",
    "    optimizer.zero_grad()\n",
    "    output = model(X_tensor)\n",
    "    loss = criterion(output, Y_tensor)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    # 每 5 epoch 记录一次\n",
    "    if epoch % 5 == 0 or epoch == epochs - 1:\n",
    "        with torch.no_grad():\n",
    "            pred_y = model(X_tensor).detach()\n",
    "            history.append(pred_y.numpy())\n",
    "\n",
    "print(\"训练完成，准备动画...\")\n",
    "\n",
    "# ========== Step 5: Matplotlib 动画可视化 ==========\n",
    "fig, ax = plt.subplots(figsize=(8, 5))\n",
    "ax.scatter(x_np, y_np, c='blue', label=\"Nonlinear Data\")\n",
    "line, = ax.plot([], [], 'r-', linewidth=2, label=\"Linear Fit\")\n",
    "text = ax.text(0.05, 0.95, '', transform=ax.transAxes, fontsize=12, verticalalignment='top')\n",
    "ax.set_xlim(-3, 3)\n",
    "ax.set_ylim(min(y_np) - 1, max(y_np) + 1)\n",
    "ax.legend()\n",
    "ax.set_title(\"Linear Model Fitting Nonlinear Data\")\n",
    "\n",
    "def init():\n",
    "    line.set_data([], [])\n",
    "    text.set_text('')\n",
    "    return line, text\n",
    "\n",
    "def animate(i):\n",
    "    y_pred = history[i].flatten()\n",
    "    line.set_data(x_np, y_pred)\n",
    "    loss_val = np.mean((y_np - y_pred)**2)\n",
    "    text.set_text(f\"Epoch: {i*5:>3}, Loss: {loss_val:.4f}\")\n",
    "    return line, text\n",
    "\n",
    "ani = animation.FuncAnimation(fig, animate, init_func=init,\n",
    "                              frames=len(history), interval=150, blit=True)\n",
    "\n",
    "plt.close()\n",
    "\n",
    "# 显示动画（在Jupyter中）\n",
    "HTML(ani.to_jshtml())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b80f3b9e-5634-4ecf-ae5d-ae4d5b958c5b",
   "metadata": {},
   "source": [
    "#### 示例3：在模型上添加ReLU激活函数并拟合非线性数据点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed8db355-9ffb-4c3e-84eb-f498347cf674",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import animation\n",
    "import numpy as np\n",
    "\n",
    "# ========== Step 1: 生成非线性数据 ==========\n",
    "np.random.seed(0)\n",
    "x_np = np.linspace(-3, 3, 200)\n",
    "noise = np.random.normal(0, 0.5, x_np.shape)\n",
    "#y_np = x_np**2 + noise  # 非线性关系示例1：幂函数\n",
    "y_np = np.sin(x_np * 2) + noise  # 非线性关系示例2：三角函数\n",
    "\n",
    "X_tensor = torch.tensor(x_np, dtype=torch.float32).view(-1, 1)\n",
    "Y_tensor = torch.tensor(y_np, dtype=torch.float32).view(-1, 1)\n",
    "\n",
    "# ========== Step 2: 定义非线性模型（带激活函数） ==========\n",
    "class NonlinearModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(1, 16),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(16, 1)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.net(x)\n",
    "\n",
    "model = NonlinearModel()\n",
    "\n",
    "# ========== Step 3: 损失函数与优化器 ==========\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.05)\n",
    "\n",
    "# ========== Step 4: 训练并记录过程 ==========\n",
    "epochs = 200\n",
    "history = []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    model.train()\n",
    "    optimizer.zero_grad()\n",
    "    output = model(X_tensor)\n",
    "    loss = criterion(output, Y_tensor)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "    if epoch % 5 == 0 or epoch == epochs - 1:\n",
    "        with torch.no_grad():\n",
    "            pred_y = model(X_tensor).detach()\n",
    "            history.append(pred_y.numpy())\n",
    "\n",
    "print(\"训练完成，准备动画...\")\n",
    "\n",
    "# ========== Step 5: 动态可视化拟合过程 ==========\n",
    "fig, ax = plt.subplots(figsize=(8, 5))\n",
    "ax.scatter(x_np, y_np, c='skyblue', label=\"Nonlinear Data\")\n",
    "line, = ax.plot([], [], 'r-', linewidth=2, label=\"Model Fit\")\n",
    "text = ax.text(0.05, 0.95, '', transform=ax.transAxes, fontsize=12, verticalalignment='top')\n",
    "ax.set_xlim(-3, 3)\n",
    "ax.set_ylim(min(y_np) - 1, max(y_np) + 1)\n",
    "ax.legend()\n",
    "ax.set_title(\"MLP with ReLU Fitting Nonlinear Data\")\n",
    "\n",
    "def init():\n",
    "    line.set_data([], [])\n",
    "    text.set_text('')\n",
    "    return line, text\n",
    "\n",
    "def animate(i):\n",
    "    y_pred = history[i].flatten()\n",
    "    line.set_data(x_np, y_pred)\n",
    "    loss_val = np.mean((y_np - y_pred)**2)\n",
    "    text.set_text(f\"Epoch: {i*5:>3}, Loss: {loss_val:.4f}\")\n",
    "    return line, text\n",
    "\n",
    "ani = animation.FuncAnimation(fig, animate, init_func=init,\n",
    "                              frames=len(history), interval=150, blit=True)\n",
    "\n",
    "plt.close()\n",
    "\n",
    "# 显示动画（在Jupyter中）\n",
    "HTML(ani.to_jshtml())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90606145-7c0c-4a50-a6cd-d4028b59a245",
   "metadata": {},
   "source": [
    "### torch.nn的层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6c4461f-3819-4a6f-a13e-63d6844b1836",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "dir(nn)  # 输出 nn 模块下所有类名"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ca62110-6cfb-403f-bcf0-01303983a0ce",
   "metadata": {},
   "source": [
    "### 损失函数示例"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4619b409-3fd3-4dc7-87d9-0e1135f7a81f",
   "metadata": {},
   "source": [
    "#### 损失函数基础应用示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ebb6630-87c8-43a6-91a8-dfd62df353f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 1. 定义损失函数\n",
    "criterion = nn.MSELoss(reduction='mean')\n",
    "\n",
    "# 2. 准备数据，手动提供input（预测值）\n",
    "input = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], requires_grad=True)\n",
    "target = torch.tensor([[1.5, 2.5, 3.5], [3.5, 4.5, 5.5]])\n",
    "\n",
    "# 3. 计算损失\n",
    "loss = criterion(input, target)\n",
    "\n",
    "# 4. 反向传播\n",
    "loss.backward()\n",
    "\n",
    "# 5. 输出损失\n",
    "print(f\"Loss (mean): {loss.item()}\")\n",
    "\n",
    "# 可选：逐元素损失\n",
    "criterion_none = nn.MSELoss(reduction='none')  # 不进行聚合，输出则为与input/target同形状的张量\n",
    "loss_none = criterion_none(input, target)\n",
    "print(f\"Element-wise loss:\\n{loss_none}\")\n",
    "\n",
    "# 验证求和损失（reduction='sum'）\n",
    "criterion_sum = nn.L1Loss(reduction='sum')\n",
    "loss_sum = criterion_sum(input, target)\n",
    "print(f\"Sum loss: {loss_sum.item()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "331fd174-7941-41bb-ac21-3c84aaa7cd4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 定义损失函数\n",
    "criterion = nn.L1Loss(reduction='mean')  # 默认均值\n",
    "\n",
    "# 示例数据\n",
    "input = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], requires_grad=True)\n",
    "target = torch.tensor([[1.5, 2.5, 3.5], [3.5, 4.5, 5.5]])\n",
    "\n",
    "# 计算损失\n",
    "loss = criterion(input, target)\n",
    "\n",
    "# 输出损失\n",
    "print(f\"Loss (mean): {loss.item()}\")  # 平均绝对误差\n",
    "\n",
    "# 反向传播\n",
    "loss.backward()\n",
    "\n",
    "# 验证逐元素损失（reduction='none'）\n",
    "criterion_none = nn.L1Loss(reduction='none')\n",
    "loss_none = criterion_none(input, target)\n",
    "print(f\"Element-wise loss:\\n{loss_none}\")\n",
    "\n",
    "# 验证求和损失（reduction='sum'）\n",
    "criterion_sum = nn.L1Loss(reduction='sum')\n",
    "loss_sum = criterion_sum(input, target)\n",
    "print(f\"Sum loss: {loss_sum.item()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4af5353-93e6-4916-b3ac-ae6589bb6cc1",
   "metadata": {},
   "source": [
    "#### Huber损失示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e311ca14-b976-4174-a3f7-fc724426a5aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 定义损失函数\n",
    "criterion = nn.SmoothL1Loss(reduction='mean', beta=1.0)\n",
    "\n",
    "# 示例数据\n",
    "input = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], requires_grad=True)\n",
    "target = torch.tensor([[1.5, 2.5, 3.5], [3.5, 4.5, 5.5]])\n",
    "\n",
    "# 计算损失\n",
    "loss = criterion(input, target)\n",
    "\n",
    "# 输出损失\n",
    "print(f\"Loss (mean): {loss.item()}\")  # 平滑 L1 损失\n",
    "\n",
    "# 反向传播\n",
    "loss.backward()\n",
    "\n",
    "# 验证逐元素损失（reduction='none'）\n",
    "criterion_none = nn.SmoothL1Loss(reduction='none', beta=1.0)\n",
    "loss_none = criterion_none(input, target)\n",
    "print(f\"Element-wise loss:\\n{loss_none}\")\n",
    "\n",
    "# 验证求和损失（reduction='sum'）\n",
    "criterion_sum = nn.SmoothL1Loss(reduction='sum', beta=1.0)\n",
    "loss_sum = criterion_sum(input, target)\n",
    "print(f\"Sum loss: {loss_sum.item()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e04e662e-2891-4860-bf5c-edf0aeb98311",
   "metadata": {},
   "source": [
    "### 分类损失函数\n",
    "\n",
    "| **损失函数**                | **输入要求**                                                                 | **目标要求**                          | **主要功能**                              | **适用场景**                                                                 | **优点**                                                                 | **局限**                                                                 | **PyTorch 实现**                     |\n",
    "|-----------------------------|-----------------------------------------------------------------------------|---------------------------------------|------------------------------------------|-----------------------------------------------------------------------------|--------------------------------------------------------------------------|--------------------------------------------------------------------------|--------------------------------------|\n",
    "| **CrossEntropyLoss**        | Logits（未经过 softmax），形状 \\([N, C, ...]\\)                               | 类别索引（整数），形状 \\([N, ...]\\)    | 结合 log-softmax 和 NLL，计算多分类交叉熵 | 单标签多分类（如图像分类、文本分类）                                         | 数值稳定，适合标准多分类任务                                              | 不支持多标签或软标签                                                     | `torch.nn.CrossEntropyLoss`          |\n",
    "| **BCEWithLogitsLoss**       | Logits（未经过 Sigmoid），形状任意                                           | 二值标签（0/1 或概率），形状与输入相同 | 结合 Sigmoid 和二元交叉熵，计算二分类损失 | 单标签/多标签二分类、图像分割、GAN 判别器                                    | 数值稳定，支持多标签和不平衡数据（通过 `pos_weight`）                      | 仅限二分类，不适合多分类                                                 | `torch.nn.BCEWithLogitsLoss`         |\n",
    "| **BCELoss**                 | 概率（经过 Sigmoid，[0, 1]），形状任意                                       | 二值标签（0/1 或概率），形状与输入相同 | 计算二元交叉熵损失                       | 二分类、多标签分类（需手动 Sigmoid）                                         | 适合已计算概率的场景                                                     | 数值稳定性稍差，需手动 Sigmoid                                           | `torch.nn.BCELoss`                   |\n",
    "| **NLLLoss**                 | Log 概率（经过 log-softmax），形状 \\([N, C, ...]\\)                           | 类别索引（整数），形状 \\([N, ...]\\)    | 计算负对数似然损失                       | 多分类（需手动 log-softmax）、语言模型                                      | 适合自定义 log-softmax 的场景                                             | 需要手动 log-softmax，不如 CrossEntropyLoss 方便                          | `torch.nn.NLLLoss`                   |\n",
    "| **KLDivLoss**               | Log 概率（通常 log-softmax 输出），形状任意                                  | 目标概率分布，形状与输入相同           | 计算 KL 散度（交叉熵的一部分）            | 分布匹配、软标签分类、知识蒸馏、VAE                                          | 适合软标签和分布优化                                                     | 需要目标概率分布，计算复杂                                               | `torch.nn.KLDivLoss`                 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db696072-ad7d-464e-ab73-12c45b29b8d9",
   "metadata": {},
   "source": [
    "#### BCELoss示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "422972fa-01e5-44fb-bb82-74318b74b1f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 模拟预测值（需要事先经过sigmoid激活）\n",
    "input = torch.tensor([0.9, 0.2, 0.8], requires_grad=True)\n",
    "\n",
    "# 真实标签（必须是0或1）\n",
    "target = torch.tensor([1.0, 0.0, 1.0])\n",
    "\n",
    "# 初始化损失函数\n",
    "criterion = nn.BCELoss()\n",
    "\n",
    "# 计算损失\n",
    "loss = criterion(input, target)\n",
    "\n",
    "print(\"BCELoss:\", loss.item())\n",
    "\n",
    "# 反向传播\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef9241eb-cf7d-46ba-9a7c-bccb3ba15613",
   "metadata": {},
   "source": [
    "#### BCEWithLogitsLoss示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11d045f2-c2ad-4ca9-8425-facfa6b67472",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# logits（未经过sigmoid）\n",
    "input = torch.tensor([0.2, -1.0, 3.0], requires_grad=True)\n",
    "\n",
    "# 标签值\n",
    "target = torch.tensor([1.0, 0.0, 1.0])\n",
    "\n",
    "# 定义损失函数\n",
    "criterion = nn.BCEWithLogitsLoss()\n",
    "\n",
    "# 计算损失\n",
    "loss = criterion(input, target)\n",
    "print(\"BCEWithLogitsLoss:\", loss.item())\n",
    "\n",
    "# 反向传播\n",
    "loss.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adf71ec6-885e-4554-922c-53e1348c8767",
   "metadata": {},
   "source": [
    "#### NLLLoss示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56cbc82d-2421-4444-a2e9-e06e56037c0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 实例化 NLLLoss，使用默认的 reduction='mean'\n",
    "nll_loss = nn.NLLLoss()\n",
    "\n",
    "# 示例 1: 简单多分类，batch_size=2\n",
    "# 模拟模型输出的原始 logits\n",
    "raw_logits = torch.tensor([\n",
    "    [0.1, 2.0, 0.3],  # 样本1\n",
    "    [1.5, 0.2, 0.8]   # 样本2\n",
    "], dtype=torch.float32)\n",
    "\n",
    "# 必须先经过LogSoftmax激活函数\n",
    "log_probabilities = torch.log_softmax(raw_logits, dim=1)\n",
    "\n",
    "# 真实类别标签 (整数索引)\n",
    "targets = torch.tensor([1, 0], dtype=torch.long) # 样本1真实类别是1，样本2真实类别是0\n",
    "\n",
    "loss = nll_loss(log_probabilities, targets)\n",
    "print(f\"NLLLoss (mean): {loss.item()}\") # 输出: NLLLoss (mean): 0.2422...\n",
    "\n",
    "# 示例 2: 使用 weight 参数处理类别不平衡\n",
    "# 假设类别0权重1.0，类别1权重10.0（少数类），类别2权重1.0\n",
    "class_weights = torch.tensor([1.0, 10.0, 1.0], dtype=torch.float32)\n",
    "nll_loss_weighted = nn.NLLLoss(weight=class_weights)\n",
    "\n",
    "# 损失计算将根据真实类别对应的权重进行加权平均\n",
    "loss_weighted = nll_loss_weighted(log_probabilities, targets)\n",
    "print(f\"NLLLoss (with class weights): {loss_weighted.item()}\") # 输出: NLLLoss (with class weights): 1.0675..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c668488c-d0eb-4df5-9a9c-636e1e2cdb91",
   "metadata": {},
   "source": [
    "#### CrossEntropyLoss示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68837d3e-a4d5-4b9b-8a5c-098bb1e2154d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 实例化 CrossEntropyLoss，使用默认的 reduction='mean'\n",
    "cross_entropy_loss = nn.CrossEntropyLoss()\n",
    "\n",
    "# 示例 1: 简单多分类，batch_size=2，3个类别\n",
    "# input 是模型输出的原始 logits (不需要Softmax)\n",
    "# 形状: (batch_size, num_classes)\n",
    "predictions_logits = torch.tensor([\n",
    "    [0.1, 2.0, 0.3],  # 样本1\n",
    "    [1.5, 0.2, 0.8]   # 样本2\n",
    "], dtype=torch.float32)\n",
    "\n",
    "# 真实类别标签，形状: (batch_size,)，是整数索引\n",
    "targets = torch.tensor([1, 0], dtype=torch.long) # 样本1真实类别是1，样本2真实类别是0\n",
    "\n",
    "loss = cross_entropy_loss(predictions_logits, targets)\n",
    "print(f\"CrossEntropyLoss (mean): {loss.item()}\") # 输出: CrossEntropyLoss (mean): 0.2422...\n",
    "\n",
    "# 示例 2: 使用 weight 参数处理类别不平衡\n",
    "# 假设类别0权重1.0，类别1权重10.0（少数类），类别2权重1.0\n",
    "class_weights = torch.tensor([1.0, 10.0, 1.0], dtype=torch.float32)\n",
    "cross_entropy_loss_weighted = nn.CrossEntropyLoss(weight=class_weights)\n",
    "\n",
    "loss_weighted = cross_entropy_loss_weighted(predictions_logits, targets)\n",
    "print(f\"CrossEntropyLoss (with class weights): {loss_weighted.item()}\") # 输出: CrossEntropyLoss (with class weights)\n",
    "\n",
    "# 示例 3: 带有ignore_index的多维输入 (例如在语义分割中)\n",
    "# 假设图像分割的logits形状为 (batch_size, num_classes, H, W)\n",
    "# target 形状为 (batch_size, H, W)，包含像素级别的类别索引\n",
    "predictions_segmentation_logits = torch.randn(2, 3, 4, 4) # 批次2，3个类别，4x4图像\n",
    "targets_segmentation = torch.randint(0, 3, (2, 4, 4), dtype=torch.long)\n",
    "# 假设类别 0 是要忽略的背景\n",
    "targets_segmentation[0, 0, 0] = 0 # 将某个位置的标签设为要忽略的索引\n",
    "targets_segmentation[1, 2, 3] = 0 # 另一个位置也设为忽略\n",
    "\n",
    "loss_fn_ignore = nn.CrossEntropyLoss(ignore_index=0) # 忽略类别0的损失\n",
    "loss_segmentation = loss_fn_ignore(predictions_segmentation_logits, targets_segmentation)\n",
    "print(f\"CrossEntropyLoss (with ignore_index): {loss_segmentation.item()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce665074-34e0-4cd6-bbc4-71d6ea3b3e56",
   "metadata": {},
   "source": [
    "### 优化器\n",
    "\n",
    "| 优化策略                     | 简介                                                                 | 特点                                                                                     | 优点                                                                 | 缺点                                                                 |\n",
    "|------------------------------|----------------------------------------------------------------------|------------------------------------------------------------------------------------------|----------------------------------------------------------------------|----------------------------------------------------------------------|\n",
    "| **标准梯度下降 (GD)**        | 使用整个数据集计算梯度，一次更新所有参数。                           | 全量数据，梯度准确，计算成本高。                                                         | 稳定，收敛可靠（凸问题）。                                           | 计算慢，内存需求大，不适合大规模数据。                               |\n",
    "| **随机梯度下降 (SGD)**       | 每次随机抽取一个样本计算梯度并更新参数。                             | 单样本更新，计算快，噪声大。                                                             | 效率高，适合大规模数据，噪声可帮助逃离局部极值。                     | 梯度波动大，收敛不稳定。                                             |\n",
    "| **小批量梯度下降 (Mini-batch GD)** | 使用小批量样本计算梯度并更新参数。                                   | 平衡稳定性和效率，batch 大小通常为 32、64 等。                                           | 兼顾效率和稳定性，广泛应用。                                         | Batch 大小需调参，可能陷入局部极值。                                 |\n",
    "| **动量法 (Momentum)**        | 引入动量项，累积历史梯度方向，加速收敛。                             | 平滑更新方向，动量参数通常为 0.9。                                                       | 加速收敛，减少震荡，适合深层网络。                                   | 需调参，可能超调。                                                   |\n",
    "| **Nesterov 加速梯度 (NAG)**  | 动量法改进，提前计算前瞻梯度。                                       | 基于“未来”位置梯度更新，更精确。                                                         | 收敛更快，适合曲率大区域。                                           | 实现复杂，需调参。                                                   |\n",
    "| **Adagrad**                  | 自适应调整学习率，根据历史梯度平方和缩放。                           | 稀疏特征更新快，频繁特征更新慢。                                                         | 适合稀疏数据，无需手动调学习率。                                     | 学习率过早衰减，可能停止更新。                                       |\n",
    "| **RMSprop**                  | Adagrad 改进，用指数移动平均平滑梯度平方累积。                       | 防止学习率过早衰减，衰减率通常为 0.9。                                                   | 适合非平稳目标，收敛稳定。                                           | 需调整初始学习率。                                                   |\n",
    "| **Adam**                     | 结合动量法和 RMSprop，维护一阶和二阶时刻的移动平均。                 | 自适应学习率，偏差校正，参数 β₁≈0.9，β₂≈0.999。                                          | 收敛快，适应性强，默认参数表现好。                                   | 泛化性能可能逊于 SGD，需调参。                                       |\n",
    "| **AdamW**                    | Adam 改进，添加权重衰减以提高泛化能力。                             | 解耦权重衰减与自适应学习率。                                                             | 泛化性能优于 Adam，适合复杂任务。                                     | 需调权重衰减系数。                                                   |\n",
    "| **AdaDelta**                 | RMSprop 扩展，无需设置全局学习率，用更新量移动平均缩放更新。         | 基于更新量移动平均调整，免去学习率设置。                                                 | 无需调学习率，适合复杂任务。                                         | 实现复杂，使用较少。                                                 |\n",
    "\n",
    "**PyTorch实现的13优化算法**\n",
    "\n",
    "| 优化算法       | 原理                                                                 | 关键参数                                                                 | 适用场景                                                   | 优点                                                                 | 缺点                                                                 |\n",
    "|----------------|----------------------------------------------------------------------|--------------------------------------------------------------------------|------------------------------------------------------------|----------------------------------------------------------------------|----------------------------------------------------------------------|\n",
    "| **SGD**        | 随机梯度下降，支持动量和 Nesterov 动量，基于小批量梯度更新。         | `lr`, `momentum`, `dampening`, `weight_decay`, `nesterov`                | 大规模数据集，需精细控制的任务（如图像分类）。              | 简单，泛化性能好，适合大规模数据，动量加速收敛。             | 需手动调学习率，收敛可能较慢，易受噪声影响。                 |\n",
    "| **Adam**       | 结合一阶和二阶动量的自适应学习率方法，收敛快。                       | `lr`, `betas`, `eps`, `weight_decay`, `amsgrad`                          | 深度学习通用任务（如 CNN、RNN、Transformer）。              | 收敛快，默认参数鲁棒，适应多种问题。                         | 泛化性能可能逊于 SGD，需调参。                               |\n",
    "| **AdamW**      | Adam 的改进，解耦权重衰减和自适应学习率，增强泛化。                 | `lr`, `betas`, `eps`, `weight_decay`                                    | 需要更好泛化的任务（如 NLP、计算机视觉）。                 | 泛化性能优于 Adam，收敛稳定。                                | 需调权重衰减系数，计算稍复杂。                               |\n",
    "| **RMSprop**    | 用指数移动平均平滑梯度平方，防止学习率过早衰减。                     | `lr`, `alpha`, `eps`, `weight_decay`, `momentum`, `centered`             | 非平稳目标（如 RNN、强化学习）。                           | 适合非平稳问题，收敛稳定。                                   | 需调初始学习率，泛化性能可能一般。                           |\n",
    "| **Adagrad**    | 自适应调整学习率，基于历史梯度平方和，适合稀疏数据。                 | `lr`, `lr_decay`, `weight_decay`, `eps`                                 | 稀疏数据任务（如推荐系统、NLP 词嵌入）。                   | 无需手动调学习率，适合稀疏梯度。                             | 学习率过早衰减，可能停止更新。                               |\n",
    "| **Adadelta**   | RMSprop 扩展，用更新量移动平均代替全局学习率。                       | `lr`, `rho`, `eps`, `weight_decay`                                      | 复杂任务，需免调学习率（如深度网络）。                     | 无需设置全局学习率，适应性强。                               | 实现复杂，使用较少，收敛可能较慢。                           |\n",
    "| **Adamax**     | Adam 变体，用梯度无穷范数代替二阶动量，适合稀疏梯度。               | `lr`, `betas`, `eps`, `weight_decay`                                    | 稀疏梯度场景（如 NLP）。                                   | 适合稀疏数据，计算简单。                                     | 应用范围较窄，泛化性能一般。                                 |\n",
    "| **NAdam**      | 结合 Adam 和 Nesterov 动量，更新更精确。                             | `lr`, `betas`, `eps`, `weight_decay`, `momentum_decay`                   | 需高精度更新的任务（如 Transformer）。                     | 收敛更快，更新方向更精确。                                   | 计算复杂度稍高，需调参。                                     |\n",
    "| **RAdam**      | Adam 的改进，通过自适应整流减少初期方差。                            | `lr`, `betas`, `eps`, `weight_decay`                                    | 需稳定初始训练的任务（如深度网络）。                       | 初期更稳定，收敛性能好。                                     | 应用较新，泛化性能需验证。                                   |\n",
    "| **ASGD**       | 对 SGD 参数更新进行平均，适合凸问题。                                | `lr`, `lambd`, `alpha`, `t0`, `weight_decay`                            | 凸优化问题（如线性回归）。                                 | 适合凸问题，收敛稳定。                                       | 深度学习中较少使用，效率较低。                               |\n",
    "| **LBFGS**      | 二阶优化，使用有限内存近似 Hessian，适合小数据集。                   | `lr`, `max_iter`, `max_eval`, `history_size`                            | 小数据集，凸优化问题。                                     | 高精度，适合简单问题。                                       | 计算密集，需 closure 函数，不适合深度学习大模型。             |\n",
    "| **SparseAdam** | Adam 的稀疏版本，优化稀疏张量梯度。                                  | `lr`, `betas`, `eps`                                                    | 稀疏梯度任务（如嵌入层优化）。                             | 适合稀疏数据，内存效率高。                                   | 仅限稀疏张量，应用受限。                                     |\n",
    "| **Rprop**      | 基于梯度符号调整步长，忽略梯度大小。                                 | `lr`, `etas`, `step_sizes`                                              | 全批量优化，简单问题。                                     | 简单，适合全批量任务。                                       | 不适合深度学习，收敛慢。                                     |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "991b1e61-0265-4992-93d8-2cfb81f687f5",
   "metadata": {},
   "source": [
    "### 使用优化器的标准步骤"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "420d3308-99c6-4a1f-9ebc-2d4020dcbd96",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "# 1. 定义模型，这里直接使用仅包含一个线性层的模型\n",
    "model = nn.Linear(10, 1)\n",
    "\n",
    "# 打印模型的初始参数\n",
    "#for name, param in model.named_parameters():\n",
    "#    print(f\"Name: {name} | Shape: {param.shape} | Value:\\n{param.data}\")\n",
    "\n",
    "# 2. 定义损失函数\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "# 3. 实例化优化器（比如SGD）\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
    "\n",
    "# 模拟训练数据\n",
    "x = torch.randn(32, 10)\n",
    "y = torch.randn(32, 1)\n",
    "\n",
    "# 4. （训练循环）为简化代码，这里仅提供一次迭代\n",
    "optimizer.zero_grad()   # 清空旧梯度\n",
    "output = model(x)       # 前向传播\n",
    "loss = criterion(output, y)  # 计算损失\n",
    "loss.backward()         # 反向传播计算梯度\n",
    "optimizer.step()        # 更新模型参数\n",
    "\n",
    "# 训练后，打印模型的参数\n",
    "#for name, param in model.named_parameters():\n",
    "#    print(f\"Name: {name} | Shape: {param.shape} | Value:\\n{param.data}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "710390e7-a7ef-4c81-8e28-db70640ddf79",
   "metadata": {},
   "source": [
    "#### 标准的模型训练流程\n",
    "- 重点在于优化器的使用步骤\n",
    "- 定义了训练循环"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0844d090-5afd-4aab-a24e-0085cc76dc82",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "\n",
    "# 1. 定义模型，通过继承nn.Module进行，仅提供了一个纯性层\n",
    "class SimpleNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SimpleNet, self).__init__()\n",
    "        self.fc = nn.Linear(10, 1)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        return self.fc(x)\n",
    "\n",
    "model = SimpleNet()\n",
    "\n",
    "# 2. 定义损失函数\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "# 3. 定义优化器\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "\n",
    "# 4. 准备数据\n",
    "X = torch.randn(100, 10)\n",
    "y = torch.randn(100, 1)\n",
    "dataset = TensorDataset(X, y)\n",
    "dataloader = DataLoader(dataset, batch_size=32, shuffle=True)\n",
    "\n",
    "# 5. 训练循环\n",
    "num_epochs = 10\n",
    "for epoch in range(num_epochs):\n",
    "    for batch_X, batch_y in dataloader:\n",
    "        optimizer.zero_grad()        # 清零梯度\n",
    "        outputs = model(batch_X)     # 前向传播\n",
    "        loss = criterion(outputs, batch_y)  # 计算损失\n",
    "        loss.backward()             # 反向传播\n",
    "        optimizer.step()            # 更新参数\n",
    "    print(f\"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}\")\n",
    "\n",
    "# 6. （可选）保存模型\n",
    "#torch.save(model.state_dict(), 'model.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b0a4584-57b5-4c10-a6d3-0d38fea6ad5e",
   "metadata": {},
   "source": [
    "### optim.SGD优化算法示例\n",
    "- 比较SGD、动量SGD和Nesterov动量的参数收敛效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21bd12b6-dd3b-4106-9b0f-c73f44074ea2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.optim as optim\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 定义要优化的函数 f(x) = (x-5)^2\n",
    "def func(x):\n",
    "    return (x - 5) ** 2\n",
    "\n",
    "# 初始化参数的起始值\n",
    "start_x = torch.tensor([0.0], requires_grad=True)\n",
    "\n",
    "# 设置优化参数\n",
    "learning_rate = 0.1\n",
    "num_iterations = 50 # 迭代次数\n",
    "\n",
    "# --- 实验一：标准 SGD (momentum = 0) ---\n",
    "print(\"--- 标准 SGD (momentum = 0) ---\")\n",
    "x_sgd = start_x.clone().detach().requires_grad_(True) # 克隆起始值，用于SGD\n",
    "optimizer_sgd = optim.SGD([x_sgd], lr=learning_rate, momentum=0)\n",
    "\n",
    "sgd_history = [x_sgd.item()]\n",
    "\n",
    "for i in range(num_iterations):\n",
    "    optimizer_sgd.zero_grad() # 清零之前的梯度\n",
    "    loss = func(x_sgd)        # 计算损失\n",
    "    loss.backward()           # 反向传播计算梯度\n",
    "    optimizer_sgd.step()      # 更新参数\n",
    "\n",
    "    sgd_history.append(x_sgd.item())\n",
    "\n",
    "    # if (i + 1) % 10 == 0:\n",
    "    #     print(f\"Iteration {i+1}: x = {x_sgd.item():.4f}, Loss = {loss.item():.4f}\")\n",
    "\n",
    "print(f\"最终参数x (标准SGD): {x_sgd.item():.4f}\")\n",
    "\n",
    "\n",
    "# --- 实验二：动量 SGD (momentum = 0.9) ---\n",
    "print(\"\\n--- 动量 SGD (momentum = 0.9) ---\")\n",
    "x_momentum = start_x.clone().detach().requires_grad_(True) # 克隆起始值，用于动量SGD\n",
    "# 注意：这里设置了 momentum 参数\n",
    "optimizer_momentum = optim.SGD([x_momentum], lr=learning_rate, momentum=0.9)\n",
    "\n",
    "momentum_history = [x_momentum.item()]\n",
    "\n",
    "for i in range(num_iterations):\n",
    "    optimizer_momentum.zero_grad() # 清零之前的梯度\n",
    "    loss = func(x_momentum)     # 计算损失\n",
    "    loss.backward()          # 反向传播计算梯度\n",
    "    optimizer_momentum.step()   # 更新参数\n",
    "\n",
    "    momentum_history.append(x_momentum.item())\n",
    "\n",
    "    # if (i + 1) % 10 == 0:\n",
    "    #     print(f\"Iteration {i+1}: x = {x_momentum.item():.4f}, Loss = {loss.item():.4f}\")\n",
    "\n",
    "print(f\"最终参数x (动量SGD): {x_momentum.item():.4f}\")\n",
    "\n",
    "# --- 实验三：Nesterov动量SGD (momentum = 0.9, nesterov = True) ---\n",
    "print(\"\\n--- Nesterov动量SGD (momentum = 0.9, nesterov = True) ---\")\n",
    "x_nesterov = start_x.clone().detach().requires_grad_(True) # 克隆起始值，用于 Nesterov SGD\n",
    "# 注意：这里设置了 momentum 和 nesterov 参数\n",
    "optimizer_nesterov = optim.SGD([x_nesterov], lr=learning_rate, momentum=0.9, nesterov=True)\n",
    "\n",
    "nesterov_history = [x_nesterov.item()]\n",
    "\n",
    "for i in range(num_iterations):\n",
    "    optimizer_nesterov.zero_grad() # 清零之前的梯度\n",
    "    loss = func(x_nesterov)     # 计算损失\n",
    "    loss.backward()          # 反向传播计算梯度\n",
    "    optimizer_nesterov.step()   # 更新参数\n",
    "\n",
    "    nesterov_history.append(x_nesterov.item())\n",
    "\n",
    "    # if (i + 1) % 10 == 0:\n",
    "    #     print(f\"Iteration {i+1}: x = {x_nesterov.item():.4f}, Loss = {loss.item():.4f}\")\n",
    "\n",
    "print(f\"最终参数x (Nesterov动量SGD): {x_nesterov.item():.4f}\")\n",
    "\n",
    "\n",
    "# --- 绘制参数随迭代次数的变化图 ---\n",
    "plt.figure(figsize=(10, 6))\n",
    "plt.plot(sgd_history, label='Standard SGD (momentum=0)', marker='o', linestyle='-')\n",
    "plt.plot(momentum_history, label='Momentum SGD (momentum=0.9)', marker='x', linestyle='--')\n",
    "plt.plot(nesterov_history, label='Nesterov Momentum SGD (momentum=0.9)', marker='*', linestyle='-.')\n",
    "plt.axhline(y=5.0, color='r', linestyle='-', label='Optimal x = 5.0') # 绘制目标值\n",
    "plt.xlabel(\"Iteration\")\n",
    "plt.ylabel(\"Parameter x value\")\n",
    "plt.title(\"Parameter x Value vs. Iteration for different SGD variants\")\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93554136-98f2-4561-9aa2-d5468503099d",
   "metadata": {},
   "source": [
    "### 自适应学习率优化算法\n",
    "\n",
    "| 特性 / 算法 | Adagrad (Adaptive Gradient) | RMSprop (Root Mean Square Propagation) | AdaDelta |\n",
    "| :---------------- | :--------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n",
    "| **核心思想** | 根据**过去所有梯度平方的累加和**来自适应地调整每个参数的学习率。 | 根据**过去梯度平方的指数加权移动平均**（Exponentially Weighted Moving Average, EWMA）来自适应地调整每个参数的学习率。 | 结合了梯度平方的 EWMA 和参数更新的 EWMA，**在大多数情况下无需手动设置学习率**。 |\n",
    "| **学习率调整** | 频繁更新的参数学习率下降更快，不频繁更新的参数学习率下降更慢。 | 解决了 Adagrad 学习率单调递减过快的问题，能更灵活地适应非稳态和非凸问题。 | 彻底消除了对全局学习率的依赖，能够动态调整学习步长。 |\n",
    "| **优点** | - 适用于**稀疏数据**（如 NLP 中的词嵌入），能有效提高低频特征的学习效率。<br>- 无需手动调整每维学习率。 | - 缓解了 Adagrad 学习率过早衰减的问题。<br>- 在循环神经网络（RNN）等任务上表现良好。<br>- 比 Adagrad 适用范围更广。 | - **无需手动设置学习率**。<br>- 训练后期学习率不会过小，有助于持续学习。<br>- 对学习率超参数不敏感。 |\n",
    "| **缺点** | - **学习率会持续单调递减**，可能导致训练后期学习率过小，模型过早停止学习。 | - 仍然需要手动设置一个**全局学习率**。<br>- 缺乏对动量的内置支持（通常需与 Momentum 结合）。 | - 相较于 Adam，在某些任务上收敛速度可能略慢。<br>- 实现相对复杂。 |\n",
    "| **主要超参数** | 初始学习率 ($\\eta$)，常数 $\\epsilon$（小常数，防止除零） | 初始学习率 ($\\eta$)，衰减率 ($\\rho$)，常数 $\\epsilon$ | 衰减率 ($\\rho$)，常数 $\\epsilon$ |\n",
    "| **数学公式** | $\\mathbf{g}_t = \\nabla L(\\theta_{t-1})$ <br>$\\mathbf{G}_t = \\mathbf{G}_{t-1} + \\mathbf{g}_t \\odot \\mathbf{g}_t$ <br>$\\theta_t = \\theta_{t-1} - \\frac{\\eta}{\\sqrt{\\mathbf{G}_t + \\epsilon}} \\odot \\mathbf{g}_t$ | $\\mathbf{g}_t = \\nabla L(\\theta_{t-1})$ <br>$\\mathbf{E}[\\mathbf{g}^2]_t = \\rho \\mathbf{E}[\\mathbf{g}^2]_{t-1} + (1-\\rho) \\mathbf{g}_t \\odot \\mathbf{g}_t$ <br>$\\theta_t = \\theta_{t-1} - \\frac{\\eta}{\\sqrt{\\mathbf{E}[\\mathbf{g}^2]_t + \\epsilon}} \\odot \\mathbf{g}_t$ | $\\mathbf{g}_t = \\nabla L(\\theta_{t-1})$ <br>$\\mathbf{E}[\\mathbf{g}^2]_t = \\rho \\mathbf{E}[\\mathbf{g}^2]_{t-1} + (1-\\rho) \\mathbf{g}_t \\odot \\mathbf{g}_t$ <br>$\\Delta \\theta_t = - \\frac{\\sqrt{\\mathbf{E}[\\Delta \\theta^2]_{t-1} + \\epsilon}}{\\sqrt{\\mathbf{E}[\\mathbf{g}^2]_t + \\epsilon}} \\odot \\mathbf{g}_t$ <br>$\\mathbf{E}[\\Delta \\theta^2]_t = \\rho \\mathbf{E}[\\Delta \\theta^2]_{t-1} + (1-\\rho) \\Delta \\theta_t \\odot \\Delta \\theta_t$ <br>$\\theta_t = \\theta_{t-1} + \\Delta \\theta_t$ |\n",
    "| **备注** | - $\\theta$: 模型参数<br>- $L$: 损失函数<br>- $\\nabla L(\\theta)$: 梯度<br>- $\\eta$: 初始学习率<br>- $\\mathbf{g}_t$: 当前梯度<br>- $\\mathbf{G}_t$: 历史梯度平方的累加和（对角矩阵）<br>- $\\mathbf{E}[\\mathbf{g}^2]_t$: 梯度平方的指数加权移动平均<br>- $\\mathbf{E}[\\Delta \\theta^2]_t$: 参数更新平方的指数加权移动平均<br>- $\\rho$: 衰减率 (RMSprop 和 Adadelta 中的超参数，通常为 0.9 或 0.99)<br>- $\\epsilon$: 极小常数，通常为 $10^{-8}$，防止分母为零<br>- $\\odot$: 元素级乘法（Hadamard积） | | |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13a63064-018b-4d10-ac9e-f1cdd5073dad",
   "metadata": {},
   "source": [
    "#### 示例：对比三个自适应学习算法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb3ca8f5-5bfa-415b-97fc-a49918fb1f58",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 设置随机种子\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# 生成数据\n",
    "X = torch.linspace(-5, 5, 1000).reshape(-1, 1)\n",
    "y = X**2 + 0.5 * X + torch.randn(X.size()) * 0.5  # y = x^2 + 0.5x + noise\n",
    "dataset = TensorDataset(X, y)\n",
    "dataloader = DataLoader(dataset, batch_size=32, shuffle=True)\n",
    "\n",
    "# 定义模型\n",
    "class SimpleNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SimpleNet, self).__init__()\n",
    "        self.layers = nn.Sequential(\n",
    "            nn.Linear(1, 64),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(64, 32),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(32, 1)\n",
    "        )\n",
    "    \n",
    "    def forward(self, x):\n",
    "        return self.layers(x)\n",
    "\n",
    "# 训练函数\n",
    "def train_model(optimizer_class, optimizer_params, num_epochs=50):\n",
    "    model = SimpleNet()\n",
    "    criterion = nn.MSELoss()\n",
    "    optimizer = optimizer_class(model.parameters(), **optimizer_params)\n",
    "    \n",
    "    losses = []\n",
    "    for epoch in range(num_epochs):\n",
    "        epoch_loss = 0\n",
    "        for batch_X, batch_y in dataloader:\n",
    "            optimizer.zero_grad()\n",
    "            outputs = model(batch_X)\n",
    "            loss = criterion(outputs, batch_y)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            epoch_loss += loss.item() * batch_X.size(0)\n",
    "        losses.append(epoch_loss / len(dataset))\n",
    "    return losses\n",
    "\n",
    "# 优化器参数\n",
    "optimizers = [\n",
    "    (optim.Adagrad, {'lr': 0.01, 'eps': 1e-10}, 'Adagrad'),\n",
    "    (optim.RMSprop, {'lr': 0.01, 'alpha': 0.99, 'eps': 1e-8}, 'RMSprop'),\n",
    "    (optim.Adadelta, {'lr': 1.0, 'rho': 0.9, 'eps': 1e-6}, 'Adadelta')\n",
    "]\n",
    "\n",
    "# 训练并记录损失\n",
    "results = {}\n",
    "for opt_class, params, name in optimizers:\n",
    "    losses = train_model(opt_class, params)\n",
    "    results[name] = losses\n",
    "\n",
    "# 可视化\n",
    "plt.figure(figsize=(10, 6))\n",
    "for name, losses in results.items():\n",
    "    plt.plot(losses, label=name)\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Adagrad, RMSprop和Adadelta的损失曲线')\n",
    "plt.legend()\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5804984-82db-4a74-8a2b-24ef241afd87",
   "metadata": {},
   "source": [
    "### 自适应动量优化算法\n",
    "\n",
    "| **优化算法** | **融合机制** | **PyTorch类名** | **关键特性** | **适用场景** |\n",
    "|--------------|--------------|------------------|--------------|--------------|\n",
    "| **Adam** | 动量法 + 自适应学习率（一阶和二阶矩的指数移动平均） | `torch.optim.Adam` | - 结合一阶动量（梯度均值）和二阶动量（梯度平方均值）<br>- 偏差修正确保早期迭代稳定性<br>- 默认参数（lr=0.001, betas=(0.9, 0.999)）鲁棒 | - 几乎所有深度学习任务（如CNN、RNN、Transformer）<br>- 非凸优化问题<br>- 通用首选优化器 |\n",
    "| **AdamW** | 动量法 + 自适应学习率 + 解耦权重衰减 | `torch.optim.AdamW` | - 改进Adam的权重衰减，解耦正则化与梯度更新<br>- 默认weight_decay=0.01，增强泛化<br>- 支持AMSGrad变体 | - 需要强正则化的任务（如Transformer、BERT）<br>- 大型模型或易过拟合场景 |\n",
    "| **AMSGrad** | 动量法 + 自适应学习率 + 二阶动量最大值 | `Adam(amsgrad=True)` 或<br> `AdamW(amsgrad=True)` | - 维护二阶动量历史最大值，增强收敛稳定性<br>- 解决Adam在某些非凸问题中的不稳定<br>- 无需单独类，通过`amsgrad=True`启用 | - 非凸优化问题<br>- Adam表现不稳定或稀疏梯度场景 |\n",
    "| **NAdam** | 动量法（Nesterov加速） + 自适应学习率 | `torch.optim.NAdam` | - 引入Nesterov动量，提前预测梯度方向<br>- 加速收敛，理论上优于Adam<br>- 增加momentum_decay参数（默认0.004） | - NLP任务（如RNN、Transformer）<br>- 需要快速收敛的场景 |\n",
    "| **RAdam** | 动量法 + 自适应学习率 + 动态二阶动量校正 | `torch.optim.RAdam` | - 校正早期二阶动量方差，减少初始不稳定性<br>- 训练初期更稳定，接近AdamW性能<br>- 默认lr=0.001 | - CV、NLP任务<br>- 需要稳定初始训练的场景 |\n",
    "| **Adamax** | 动量法 + 自适应学习率（无穷范数） | `torch.optim.Adamax` | - 使用梯度无穷范数代替二阶动量<br>- 对稀疏梯度或噪声大场景更稳定<br>- 默认lr=0.002，计算更简单 | - 稀疏数据（如NLP嵌入层）<br>- 噪声较大的优化任务 |\n",
    "| **SparseAdam** | 动量法 + 自适应学习率（稀疏优化） | `torch.optim.SparseAdam` | - 专为稀疏梯度设计，仅更新非零梯度<br>- 节省计算和内存<br>- 不支持weight_decay | - 稀疏梯度场景（如nn.Embedding(sparse=True)）<br>- NLP中的词嵌入优化 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6df2ca38-302c-4bba-b78c-ec54e77d26d4",
   "metadata": {},
   "source": [
    "### 数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87f1ebaf-00ae-4f38-8fa4-74959d1eb827",
   "metadata": {},
   "source": [
    "#### Dataset示例\n",
    "以下是一个简单的自定义Dataset示例，假设数据集包含图像和标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89af7924-0210-4a8d-964f-a07df71f772d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import Dataset\n",
    "import numpy as np\n",
    "\n",
    "class SimpleDataset(Dataset):\n",
    "    def __init__(self, data, labels, transform=None):\n",
    "        \"\"\"\n",
    "        初始化数据集\n",
    "        Args:\n",
    "            data: 特征数据（如图像或数值数组）\n",
    "            labels: 对应的标签\n",
    "            transform: 数据增强或预处理函数\n",
    "        \"\"\"\n",
    "        self.data = data  # 假设为NumPy数组\n",
    "        self.labels = labels\n",
    "        self.transform = transform\n",
    "\n",
    "    def __len__(self):\n",
    "        \"\"\"返回数据集大小\"\"\"\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        \"\"\"根据索引返回单个样本\"\"\"\n",
    "        sample = self.data[index]\n",
    "        label = self.labels[index]\n",
    "\n",
    "        # 应用数据增强或预处理\n",
    "        if self.transform:\n",
    "            sample = self.transform(sample)\n",
    "\n",
    "        # 转换为 PyTorch Tensor\n",
    "        sample = torch.tensor(sample, dtype=torch.float32)\n",
    "        label = torch.tensor(label, dtype=torch.long)\n",
    "        return sample, label"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c0248d1-8d92-4924-9ae0-e083af7cfe1d",
   "metadata": {},
   "source": [
    "#### 引用Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddc42d55-a74f-46c8-a95b-11994e0ab504",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 创建虚拟数据\n",
    "data = np.random.randn(100, 3, 32, 32)  # 模拟生成100张3x32x32的图像\n",
    "labels = np.random.randint(0, 10, size=100)  # 10个标签 (0-9)\n",
    "dataset = SimpleDataset(data, labels)\n",
    "\n",
    "# 访问数据集\n",
    "sample, label = dataset[0]\n",
    "print(f\"Sample shape: {sample.shape}, Label: {label}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c59f5d42-7fd7-45b7-bc42-cb3d248b47ca",
   "metadata": {},
   "source": [
    "#### 由DataLoader加载数据\n",
    "\n",
    "**关键参数**\n",
    "- batch_size：每批样本数，需根据GPU内存调整\n",
    "- shuffle：训练时设为True，验证/测试时设为 False。\n",
    "- num_workers：多线程加载，建议4-8（视CPU核心数）。\n",
    "- pin_memory：启用以加速GPU数据传输。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdb1151a-c410-4218-aad6-90168ae51c42",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "# 创建 DataLoader\n",
    "dataloader = DataLoader(\n",
    "    dataset,\n",
    "    batch_size=16,      # 批次大小\n",
    "    shuffle=True,       # 打乱数据\n",
    "    num_workers=4,      # 多线程加载\n",
    "    #pin_memory=True     # 加速GPU传输\n",
    ")\n",
    "\n",
    "# 遍历数据\n",
    "for batch_data, batch_labels in dataloader:\n",
    "    print(f\"Batch data shape: {batch_data.shape}, Batch labels shape: {batch_labels.shape}\")\n",
    "    # 送入模型训练..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5140d3c2-7bbb-4f19-94cc-000ba51e3bf2",
   "metadata": {},
   "source": [
    "#### 完整的极简示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "326b3d80-8523-49ac-ad14-93f988816ee2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "# Step 1: 定义一个简单的数据集（继承 Dataset）\n",
    "class MyDataset(Dataset):\n",
    "    def __init__(self):\n",
    "        # 随机生成 100 个样本，每个样本是 10 维特征向量\n",
    "        self.data = torch.randn(100, 10)\n",
    "        # 随机生成标签（0 或 1）\n",
    "        self.labels = torch.randint(0, 2, (100,))\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        x = self.data[idx]\n",
    "        y = self.labels[idx]\n",
    "        return x, y\n",
    "\n",
    "# Step 2: 实例化数据集\n",
    "dataset = MyDataset()\n",
    "\n",
    "# Step 3: 用 DataLoader 封装数据集\n",
    "dataloader = DataLoader(dataset, batch_size=8, shuffle=True)\n",
    "\n",
    "# Step 4: 模拟训练过程中读取数据\n",
    "for batch_idx, (features, labels) in enumerate(dataloader):\n",
    "    print(f\"Batch {batch_idx + 1}\")\n",
    "    print(f\"  Features: {features.shape}\")  # 应该是 [8, 10]\n",
    "    print(f\"  Labels:   {labels}\")\n",
    "    \n",
    "    # 模拟一个训练步骤\n",
    "    # output = model(features)\n",
    "    # loss = loss_fn(output, labels)\n",
    "    # loss.backward()\n",
    "    # optimizer.step()\n",
    "\n",
    "    if batch_idx == 2:  # 演示用：只打印前 3 个批次\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b97f7da7-3dc9-469c-9c41-78afdea14a0a",
   "metadata": {},
   "source": [
    "#### 使用自定义义collate_fn处理变长序列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7a1e63a-8200-4172-b9f8-c1e045724e7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "# --- 1. 定义 TextDataset (来自上一个教程) ---\n",
    "class TextDataset(Dataset):\n",
    "    def __init__(self, texts, labels):\n",
    "        # 假设文本已转换为ID序列的Tensor\n",
    "        self.texts = [torch.tensor(t, dtype=torch.long) for t in texts]\n",
    "        self.labels = labels\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.texts)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.texts[idx], self.labels[idx]\n",
    "\n",
    "# --- 2. 定义自定义 collate_fn ---\n",
    "def custom_collate_fn_for_padding(batch):\n",
    "    \"\"\"\n",
    "    自定义 collate_fn 来处理变长序列填充。\n",
    "    Args:\n",
    "        batch (list): 包含 (text_tensor, label) 元组的列表。\n",
    "                      例如：[(tensor([1,2,3]), 0), (tensor([4,5]), 1)]\n",
    "    Returns:\n",
    "        tuple: 包含填充后的文本批次和标签批次的元组。\n",
    "    \"\"\"\n",
    "    texts, labels = zip(*batch) # 解包批次，得到文本列表和标签列表\n",
    "\n",
    "    # 获取批次中最长的文本序列长度\n",
    "    max_len = max([len(t) for t in texts])\n",
    "\n",
    "    # 对所有文本序列进行填充，使其达到 max_len\n",
    "    # 填充值通常选择0或其他特殊token_id\n",
    "    padded_texts = []\n",
    "    for t in texts:\n",
    "        padded_text = torch.cat([t, torch.zeros(max_len - len(t), dtype=t.dtype)])\n",
    "        padded_texts.append(padded_text)\n",
    "\n",
    "    # 将填充后的文本序列堆叠成一个批次Tensor\n",
    "    texts_batch = torch.stack(padded_texts)\n",
    "    labels_batch = torch.tensor(labels, dtype=torch.long)\n",
    "\n",
    "    return texts_batch, labels_batch\n",
    "\n",
    "# --- 3. 准备模拟变长文本数据 ---\n",
    "mock_texts = [\n",
    "    [101, 200, 301, 405],\n",
    "    [101, 502],\n",
    "    [101, 603, 704, 805, 906, 107],\n",
    "    [101]\n",
    "] # 假设是词ID序列，101是CLS token\n",
    "mock_text_labels = [0, 1, 0, 1]\n",
    "\n",
    "# --- 4. 创建 Dataset 实例 ---\n",
    "text_dataset = TextDataset(mock_texts, mock_text_labels)\n",
    "\n",
    "# --- 5. 创建 DataLoader 实例，并传入 custom_collate_fn_for_padding ---\n",
    "text_loader = DataLoader(\n",
    "    dataset=text_dataset,\n",
    "    batch_size=2,\n",
    "    shuffle=False, # 为了方便观察填充效果，不洗牌\n",
    "    collate_fn=custom_collate_fn_for_padding # 使用自定义的合并函数\n",
    ")\n",
    "\n",
    "# --- 6. 迭代 DataLoader 并观察输出 ---\n",
    "print(\"\\n--- Using DataLoader with custom collate_fn for variable-length sequences ---\")\n",
    "for batch_idx, (text_batch, label_batch) in enumerate(text_loader):\n",
    "    print(f\"\\nBatch {batch_idx}:\")\n",
    "    print(f\"  Text batch shape: {text_batch.shape}\")\n",
    "    print(f\"  Text batch content:\\n{text_batch}\")\n",
    "    print(f\"  Label batch: {label_batch.tolist()}\")\n",
    "\n",
    "    # 预期输出：\n",
    "    # Batch 0: (最大长度为 4)\n",
    "    #   Text batch shape: torch.Size([2, 4])\n",
    "    #   Text batch content:\n",
    "    #   tensor([[101, 200, 301, 405],\n",
    "    #           [101, 502,   0,   0]])\n",
    "    #   Label batch: [0, 1]\n",
    "    #\n",
    "    # Batch 1: (最大长度为 6)\n",
    "    #   Text batch shape: torch.Size([2, 6])\n",
    "    #   Text batch content:\n",
    "    #   tensor([[101, 603, 704, 805, 906, 107],\n",
    "    #           [101,   0,   0,   0,   0,   0]])\n",
    "    #   Label batch: [0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21d25f72-554b-4c62-addc-94256539d62a",
   "metadata": {},
   "source": [
    "#### 使用torchvision.datasets示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6270605-f03c-4d1b-bfbe-b8c7f3fa5463",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "# 定义数据增强\n",
    "transform = transforms.Compose([\n",
    "    transforms.ToTensor(),  # 转换为 Tensor\n",
    "    transforms.Normalize((0.1307,), (0.3081,))  # MNIST 均值和标准差\n",
    "])\n",
    "\n",
    "# 加载数据集\n",
    "train_dataset = torchvision.datasets.MNIST(\n",
    "    root='./data',      # 数据存储路径\n",
    "    train=True,         # 训练集\n",
    "    download=True,      # 自动下载\n",
    "    transform=transform  # 数据增强\n",
    ")\n",
    "\n",
    "test_dataset = torchvision.datasets.MNIST(\n",
    "    root='./data',\n",
    "    train=False,\n",
    "    download=True,\n",
    "    transform=transform\n",
    ")\n",
    "\n",
    "# 创建 DataLoader\n",
    "train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True, num_workers=2)\n",
    "test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False, num_workers=2)\n",
    "\n",
    "# 遍历数据\n",
    "for images, labels in train_loader:\n",
    "    print(f\"Images shape: {images.shape}, Labels shape: {labels.shape}\")\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d68448d6-d7b7-437a-bbcd-698bd18d601d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## MLP示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfad85f5-4d65-4b68-bc89-ec318af8e496",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data import DataLoader\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "# 设备配置\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# 定义 MLP 模型\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self, input_size=784, hidden_size=256, output_size=10):\n",
    "        super(MLP, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(hidden_size, hidden_size)\n",
    "        self.fc3 = nn.Linear(hidden_size, output_size)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = x.view(-1, 784)  # 展平输入 (batch_size, 784)\n",
    "        x = self.relu(self.fc1(x))\n",
    "        x = self.relu(self.fc2(x))\n",
    "        x = self.fc3(x)  # 输出层不加激活（由损失函数处理）\n",
    "        return x\n",
    "\n",
    "# 数据准备\n",
    "transform = transforms.ToTensor()\n",
    "train_dataset = datasets.MNIST(root='./data', train=True, transform=transform, download=True)\n",
    "test_dataset = datasets.MNIST(root='./data', train=False, transform=transform)\n",
    "train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)\n",
    "test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)\n",
    "\n",
    "# 初始化模型、损失函数和优化器\n",
    "model = MLP().to(device)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
    "\n",
    "# 训练循环\n",
    "num_epochs = 5\n",
    "for epoch in range(num_epochs):\n",
    "    model.train()\n",
    "    running_loss = 0.0\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "        \n",
    "        optimizer.zero_grad()  # 清空梯度\n",
    "        outputs = model(images)  # 前向传播\n",
    "        loss = criterion(outputs, labels)  # 计算损失\n",
    "        loss.backward()  # 反向传播\n",
    "        optimizer.step()  # 更新参数\n",
    "        running_loss += loss.item()\n",
    "    \n",
    "    print(f\"Epoch {epoch+1}, Loss: {running_loss / len(train_loader):.4f}\")\n",
    "    \n",
    "    # 测试\n",
    "    model.eval()\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    with torch.no_grad():\n",
    "        for images, labels in test_loader:\n",
    "            images, labels = images.to(device), labels.to(device)\n",
    "            outputs = model(images)\n",
    "            _, predicted = torch.max(outputs, 1)\n",
    "            total += labels.size(0)\n",
    "            correct += (predicted == labels).sum().item()\n",
    "    \n",
    "    print(f\"Test Accuracy: {100 * correct / total:.2f}%\")\n",
    "\n",
    "# 保存模型\n",
    "torch.save(model.state_dict(), 'mlp_mnist.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7d01686-1633-4718-8f53-30d905f1c82e",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## 模型参数初始化\n",
    "\n",
    "下面用PyTorch演示一个完整的“参数初始化”的代码流程，内容包括：\n",
    "- 构建一个简单的神经网络模型（包含Linear和BatchNorm层）\n",
    "- 查看默认的初始化结果\n",
    "- 自定义初始化方法（如Xavier和Kaiming）\n",
    "- 应用自定义初始化并验证效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b6f1132-bcb1-4118-b4ba-4499027d0cb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 第一步：导入库\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 设置随机种子以确保结果可重复\n",
    "torch.manual_seed(42)\n",
    "\n",
    "# 第二步：定义一个简单的MLP模型\n",
    "class SimpleMLP(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SimpleMLP, self).__init__()\n",
    "        self.fc1 = nn.Linear(10, 32)      # 输入层 -> 隐藏层\n",
    "        self.bn1 = nn.BatchNorm1d(32)     # 批归一化\n",
    "        self.relu = nn.ReLU()             # 激活函数\n",
    "        self.fc2 = nn.Linear(32, 2)       # 隐藏层 -> 输出层（2类）\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.bn1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "# 实例化模型\n",
    "model = SimpleMLP()\n",
    "\n",
    "# 第三步：查看默认初始化的参数分布\n",
    "print(\"默认初始化参数（均值 & 标准差）:\")\n",
    "for name, param in model.named_parameters():\n",
    "    #print(name, param)\n",
    "    if 'weight' in name or 'bias' in name:\n",
    "        print(f\"{name:20s} | mean = {param.mean():.4f}, std = {param.std():.4f}\")\n",
    "\n",
    "# 第四步：定义自定义初始化函数\n",
    "def custom_init(m):\n",
    "    if isinstance(m, nn.Linear):\n",
    "        nn.init.xavier_uniform_(m.weight)     # 使用 Xavier 均匀初始化权重\n",
    "        # 说明：可用以下三种初始化方法替代上面的初始化方法，进行对比测试\n",
    "        # nn.init.kaiming_normal_(m.weight, nonlinearity='relu')  # Kaiming初始化（适用于ReLU激活）\n",
    "        # nn.init.normal_(m.weight, mean=0.0, std=0.02)  # 正态分布\n",
    "        # nn.init.uniform_(m.weight, a=-0.1, b=0.1)   # 均匀分布\n",
    "        nn.init.constant_(m.bias, 0)          # 偏置初始化为 0\n",
    "    elif isinstance(m, nn.BatchNorm1d):\n",
    "        nn.init.constant_(m.weight, 1)        # γ = 1\n",
    "        nn.init.constant_(m.bias, 0)          # β = 0\n",
    "\n",
    "# 应用自定义初始化\n",
    "model.apply(custom_init)\n",
    "\n",
    "# 第五步：查看初始化后的参数分布\n",
    "print(\"\\n自定义初始化参数（均值 & 标准差）:\")\n",
    "for name, param in model.named_parameters():\n",
    "    if 'weight' in name or 'bias' in name:\n",
    "        print(f\"{name:20s} | mean = {param.mean():.4f}, std = {param.std():.4f}\")\n",
    "\n",
    "# 第六步（可选）：验证模型输出是否正常\n",
    "# 构造一个随机输入batch（大小为4，特征维度为10）\n",
    "x = torch.randn(4, 10)\n",
    "output = model(x)\n",
    "\n",
    "print(\"\\n模型输出:\")\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3606ba3a-2459-45af-8a4b-dc5fa88355a8",
   "metadata": {},
   "source": [
    "### 模型中需要进行初始化的位置\n",
    "\n",
    "| 层类型                     | 需要初始化的参数                                                                 | 常见初始化方法                                                                 |\n",
    "|----------------------------|----------------------------------------------------------------------------------|----------------------------------------------------------------------------------|\n",
    "| 全连接层                   | 权重矩阵 \\( W \\)（形状：$[n_{\\text{in}}, n_{\\text{out}}]$）<br>偏置向量 \\( b \\)（形状：$[n_{\\text{out}}]$） | 权重：Xavier（tanh/sigmoid）、He（ReLU）、正交初始化<br>偏置：零初始化或小常数（如0.01） |\n",
    "| 卷积层                     | 卷积核（权重）（形状：$[k_h, k_w, c_{\\text{in}}, c_{\\text{out}}]$）<br>偏置向量（形状：$[c_{\\text{out}}]$） | 权重：He（ReLU）、Xavier<br>偏置：零初始化                                       |\n",
    "| 循环神经网络（RNN/LSTM/GRU） | 权重矩阵（如 $ W_{xh}, W_{hh} $，形状依层设计）<br>偏置向量（形状：$[n_{\\text{hidden}}]$） | 权重：正交初始化、Xavier<br>偏置：零初始化，LSTM遗忘门偏置可为1                |\n",
    "| 注意力机制（Attention）      | 权重矩阵（如 $ W_Q, W_K, W_V $，形状：$[d_{\\text{model}}, d_k/d_v]$）<br>偏置向量（部分实现） | 权重：Xavier、He<br>偏置：零初始化                                              |\n",
    "| 嵌入层（Embedding）         | 嵌入矩阵（形状：$[n_{\\text{vocab}}, d_{\\text{emb}}]$）                          | 随机初始化（均匀/正态分布）、预训练嵌入（如GloVe）、正弦/余弦位置编码            |\n",
    "| 批归一化（Batch Norm）      | 缩放参数 $ \\gamma $（通常为1）<br>偏移参数 $ \\beta $（通常为0）                | $ \\gamma \\sim 1 $，$ \\beta \\sim 0 $                                         |\n",
    "\n",
    "**备注**：\n",
    "- 池化层、激活函数、Dropout层通常无需初始化参数。\n",
    "- 初始化方法需根据激活函数（如ReLU、tanh）和网络深度选择。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55ab3c65-50a3-4cb3-9a2b-b02e2f0359d9",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## 正则化技术\n",
    "\n",
    "| 正则化技术 | 定义 | 作用 | 优点 | 缺点 | 应用场景 |\n",
    "|------------|------|------|------|------|----------|\n",
    "| **L1正则化** | 在损失函数中加入参数的L1范数（绝对值和） | 产生稀疏解，实现特征选择，降低模型复杂度 | 1. 稀疏模型便于解释<br>2. 适合高维数据 | 1. 非光滑优化问题<br>2. 对噪声敏感 | 线性模型（如Lasso回归）、稀疏神经网络 |\n",
    "| **L2正则化** | 在损失函数中加入参数的L2范数（平方和） | 平滑权重，降低对噪声的敏感性 | 1. 优化光滑，便于梯度下降<br>2. 增强模型稳定性 | 1. 不产生稀疏解<br>2. 需调整正则化强度 | 线性回归、逻辑回归、神经网络（权重衰减） |\n",
    "| **Dropout** | 训练时随机丢弃部分神经元 | 防止过度依赖某些神经元，模拟模型集成 | 1. 简单高效<br>2. 减少过拟合 | 1. 增加训练时间<br>2. 测试时需权重调整 | 深度神经网络（如CNN、RNN） |\n",
    "| **数据增强** | 对训练数据进行变换（如旋转、翻转） | 增加数据多样性，提高泛化能力 | 1. 不改变模型结构<br>2. 适合视觉/语音任务 | 1. 需设计合适策略<br>2. 增加预处理成本 | 图像分类、语音处理、NLP |\n",
    "| **早停** | 验证集误差不再下降时停止训练 | 限制训练轮数，防止过拟合 | 1. 简单易实现<br>2. 无需改模型 | 1. 需划分验证集<br>2. 可能导致欠拟合 | 深度学习、梯度提升树 |\n",
    "| **批标准化** | 规范化每层输入的均值和方差 | 减少内部协变量偏移，稳定训练 | 1. 加速收敛<br>2. 减少对Dropout依赖 | 1. 小批量效果差<br>2. 增加计算开销 | 深度神经网络（CNN、全连接网络） |\n",
    "| **弹性网正则化** | 结合L1和L2正则化 | 综合稀疏性和平滑性 | 1. 兼具特征选择和平滑<br>2. 适合复杂特征相关性 | 1. 需调两个超参数<br>2. 增加调参难度 | 高维特征回归、稀疏线性模型 |\n",
    "| **噪声注入** | 在数据/权重中添加随机噪声 | 增强模型鲁棒性 | 1. 简单易实现<br>2. 提高抗噪能力 | 1. 需调整噪声水平<br>2. 可能破坏数据结构 | 语音识别、图像处理、时间序列 |\n",
    "| **参数共享** | 强制模型共享部分参数（如卷积核） | 减少自由参数，降低复杂度 | 1. 降低计算量<br>2. 适合结构化数据 | 1. 仅限特定模型<br>2. 适用范围有限 | 卷积神经网络、循环神经网络 |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0616aec-e8f0-4238-917e-0a7d02df0513",
   "metadata": {},
   "source": [
    "### L1和L2正则化示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe429c62-4bff-4008-bc2c-f08ebf9b46c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "# 定义简单模型\n",
    "model = nn.Linear(10, 1)\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01, weight_decay=0.01)  # L2 正则化（权重衰减）\n",
    "\n",
    "# 训练数据\n",
    "x = torch.randn(32, 10)\n",
    "y = torch.randn(32, 1)\n",
    "\n",
    "# 前向传播\n",
    "output = model(x)\n",
    "loss = criterion(output, y)\n",
    "\n",
    "# 手动添加 L1 正则化\n",
    "l1_lambda = 0.01\n",
    "l1_norm = sum(p.abs().sum() for p in model.parameters())  # 计算L1范数\n",
    "loss += l1_lambda * l1_norm\n",
    "\n",
    "# 反向传播和优化\n",
    "optimizer.zero_grad()\n",
    "loss.backward()\n",
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92120618-5ce8-4210-8c09-d07e930a84cf",
   "metadata": {},
   "source": [
    "### Dropout示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52fba890-1595-469b-a6cb-ff237ec0a566",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 定义带 Dropout 的模型\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.fc1 = nn.Linear(10, 20)\n",
    "        self.dropout = nn.Dropout(p=0.5)  # 50% 概率丢弃\n",
    "        self.fc2 = nn.Linear(20, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = self.dropout(x)  # 应用 Dropout\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "# 示例\n",
    "model = Net()\n",
    "model.train()  # 启用 Dropout\n",
    "x = torch.randn(32, 10)\n",
    "output = model(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59596db2-9965-44ec-a0ff-540b2c9ae861",
   "metadata": {},
   "source": [
    "### Batch Normalization示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b213aa06-2cf6-4c7e-94a5-676e082f7d9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 定义带批标准化的模型\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv = nn.Conv2d(3, 16, kernel_size=3, padding=1)\n",
    "        self.bn = nn.BatchNorm2d(16)  # 批标准化，16个通道\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc = nn.Linear(16 * 32 * 32, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.conv(x)\n",
    "        x = self.bn(x)  # 应用批标准化\n",
    "        x = self.relu(x)\n",
    "        x = x.view(x.size(0), -1)\n",
    "        x = self.fc(x)\n",
    "        return x\n",
    "\n",
    "# 示例\n",
    "model = Net()\n",
    "model.train()  # 启用批标准化\n",
    "x = torch.randn(32, 3, 32, 32)\n",
    "output = model(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "542700a2-b908-49cb-9677-88d331dc2a03",
   "metadata": {},
   "source": [
    "### 早停法示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a741581-73ae-4548-a33f-451f89e78815",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "# 定义模型和数据\n",
    "model = nn.Linear(10, 1)\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)\n",
    "train_loader = ...  # 假设已有训练和验证数据加载器\n",
    "val_loader = ...\n",
    "\n",
    "# 提前停止参数\n",
    "patience = 10\n",
    "best_val_loss = float('inf')\n",
    "counter = 0\n",
    "best_model_state = None\n",
    "\n",
    "# 训练循环\n",
    "for epoch in range(100):\n",
    "    model.train()\n",
    "    for x, y in train_loader:\n",
    "        optimizer.zero_grad()\n",
    "        output = model(x)\n",
    "        loss = criterion(output, y)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "    # 验证\n",
    "    model.eval()\n",
    "    val_loss = 0\n",
    "    with torch.no_grad():\n",
    "        for x, y in val_loader:\n",
    "            output = model(x)\n",
    "            val_loss += criterion(output, y).item()\n",
    "    val_loss /= len(val_loader)\n",
    "\n",
    "    # 提前停止逻辑\n",
    "    if val_loss < best_val_loss:\n",
    "        best_val_loss = val_loss\n",
    "        best_model_state = model.state_dict()\n",
    "        counter = 0\n",
    "    else:\n",
    "        counter += 1\n",
    "        if counter >= patience:\n",
    "            print(f\"Early stopping at epoch {epoch}\")\n",
    "            break\n",
    "\n",
    "# 加载最佳模型\n",
    "model.load_state_dict(best_model_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfb82c27-ddde-4464-9c3e-8fb291b4e9ef",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## 模型保存与加载"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3e1517d-da0b-4df8-9cf1-24c0a4285de7",
   "metadata": {},
   "source": [
    "### 查看state_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "791b5e2a-78e5-471a-a8ad-8767ebac4bbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 定义一个简单的前馈神经网络 (FNN)\n",
    "class SimpleFNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(SimpleFNN, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size) # 全连接层 1\n",
    "        self.relu = nn.ReLU()                         # 激活函数 (没有参数)\n",
    "        self.fc2 = nn.Linear(hidden_size, output_size) # 全连接层 2\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "# 实例化模型\n",
    "input_size = 10\n",
    "hidden_size = 20\n",
    "output_size = 3\n",
    "model = SimpleFNN(input_size, hidden_size, output_size)\n",
    "\n",
    "# 获取模型的 state_dict\n",
    "model_state_dict = model.state_dict()\n",
    "\n",
    "# 打印 state_dict 的类型\n",
    "print(f\"State dict type: {type(model_state_dict)}\")\n",
    "\n",
    "# 打印 state_dict 中的键和对应的张量形状\n",
    "print(\"\\nKeys and shapes in state_dict:\")\n",
    "for key, value in model_state_dict.items():\n",
    "    print(f\"Key: {key}, Shape: {value.shape}\")\n",
    "\n",
    "# 打印其中一个参数的张量内容 (仅部分展示)\n",
    "print(f\"\\nContent of '{list(model_state_dict.keys())[0]}':\")\n",
    "print(model_state_dict[list(model_state_dict.keys())[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "975eaac6-4ec2-4d45-b048-410849b62f71",
   "metadata": {},
   "source": [
    "### 查看持续缓冲区\n",
    "\n",
    "给上面的示例添加一个BatchNorm层，state_dict字典中在可学习参数之外，还会多出持久缓冲区。\n",
    "\n",
    "下面示例中的bn1层不仅有可学习参数weight和bias（对应BatchNorm的$\\gamma$和$\\beta$参数，它们是可学习的），还有三个缓冲区：running_mean、running_var和num_batches_tracked。这些缓冲区是模型状态的重要组成部分，在推理时需要使用，因此也包含在state_dict中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29df4b0f-a310-4152-b6d9-845c8fafa460",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class FNNWithBatchNorm(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(FNNWithBatchNorm, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        # BatchNorm 1D 常用于全连接层之后\n",
    "        self.bn1 = nn.BatchNorm1d(hidden_size)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(hidden_size, output_size)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.bn1(x) # 应用 BatchNorm\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "# 实例化模型\n",
    "input_size = 10\n",
    "hidden_size = 20\n",
    "output_size = 3\n",
    "model_bn = FNNWithBatchNorm(input_size, hidden_size, output_size)\n",
    "\n",
    "# 获取模型的 state_dict\n",
    "model_bn_state_dict = model_bn.state_dict()\n",
    "\n",
    "# 打印 state_dict 中的键和对应的张量形状\n",
    "print(\"\\nKeys and shapes in state_dict (with BatchNorm):\")\n",
    "for key, value in model_bn_state_dict.items():\n",
    "    print(f\"Key: {key}, Shape: {value.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94c216b9-067c-4bab-9342-9a83305bf856",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### 保存和加载state_dict的完整的示例\n",
    "\n",
    "#### 保存state_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf247d46-84d8-4f4f-96e8-2a840cc606eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import os # 用于检查文件是否存在\n",
    "\n",
    "# 假设我们已经定义并（可能）训练了一个 SimpleFNN 模型实例\n",
    "# （这里我们为了演示，只实例化，不进行实际训练）\n",
    "\n",
    "class SimpleFNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(SimpleFNN, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(hidden_size, output_size)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "# 实例化模型\n",
    "input_size = 10\n",
    "hidden_size = 20\n",
    "output_size = 3\n",
    "model = SimpleFNN(input_size, hidden_size, output_size)\n",
    "\n",
    "# 定义保存文件的路径\n",
    "model_params_path = 'simple_fnn_params.pth' # 常用的文件扩展名是 .pt 或 .pth\n",
    "\n",
    "# --- 保存模型的 state_dict ---\n",
    "print(f\"Saving model state_dict to {model_params_path}...\")\n",
    "try:\n",
    "    # 获取模型的 state_dict\n",
    "    state_dict_to_save = model.state_dict()\n",
    "\n",
    "    # 使用 torch.save() 保存 state_dict\n",
    "    torch.save(state_dict_to_save, model_params_path)\n",
    "\n",
    "    print(\"Model state_dict saved successfully!\")\n",
    "    print(f\"Checking if file exists: {os.path.exists(model_params_path)}\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"An error occurred while saving: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dca6b4d0-5cf4-4241-b0c5-ba569bbf9765",
   "metadata": {},
   "source": [
    "#### 加载state_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0a5ad36-ed0a-42dd-b287-733007d2f108",
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- 加载 state_dict ---\n",
    "print(f\"\\nLoading state_dict from {model_params_path}...\")\n",
    "try:\n",
    "    # 使用 torch.load() 加载 state_dict\n",
    "    loaded_state_dict = torch.load(model_params_path)\n",
    "\n",
    "    print(\"State dict loaded successfully!\")\n",
    "    print(f\"Type of loaded object: {type(loaded_state_dict)}\")\n",
    "\n",
    "    # 我们可以像查看原始 state_dict 一样查看加载的字典\n",
    "    print(\"\\nKeys in loaded state_dict:\")\n",
    "    for key in loaded_state_dict.keys():\n",
    "        print(key)\n",
    "\n",
    "except FileNotFoundError:\n",
    "    print(f\"Error: The file '{model_params_path}' was not found.\")\n",
    "except Exception as e:\n",
    "    print(f\"An error occurred while loading: {e}\")\n",
    "\n",
    "# --- 将加载的 state_dict 载入到新的模型实例 ---\n",
    "print(\"\\nLoading state_dict into a new model instance...\")\n",
    "\n",
    "# !!! 关键步骤 1: 创建一个新的模型实例 !!!\n",
    "# 这个实例的结构 (层、层的大小、层之间的连接) 必须与保存 state_dict 时使用的模型一致\n",
    "new_model = SimpleFNN(input_size, hidden_size, output_size)\n",
    "print(\"New model instance created.\")\n",
    "\n",
    "try:\n",
    "    # !!! 关键步骤 2: 将加载的state_dict载入到新模型实例中 !!!\n",
    "    # load_state_dict 将字典中的参数复制到模型的对应位置\n",
    "    # 默认情况下 (strict=True), 加载的state_dict的键必须精确匹配模型实例的state_dict\n",
    "    new_model.load_state_dict(loaded_state_dict)\n",
    "\n",
    "    print(\"State dict loaded into new model instance successfully!\")\n",
    "\n",
    "    # 加载完成后，新模型的参数就和保存时的模型参数完全一样了\n",
    "\n",
    "except RuntimeError as e:\n",
    "    # 如果模型结构不匹配，load_state_dict会抛出RuntimeError\n",
    "    print(f\"Error loading state dict: Model structure mismatch. Details:\\n{e}\")\n",
    "except Exception as e:\n",
    "    print(f\"An unexpected error occurred: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e6069f7-767c-4b01-9046-518f19f43d51",
   "metadata": {},
   "source": [
    "#### 推理模式的特殊设定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8006433-9506-4e2c-b9e5-e903fe192be4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 继续上面的例子...\n",
    "\n",
    "# 将加载参数后的模型设置为评估模式\n",
    "new_model.eval()\n",
    "print(\"\\nModel set to evaluation mode.\")\n",
    "\n",
    "# 模拟一个输入张量 (通常是一个 batch 的数据)\n",
    "dummy_input = torch.randn(1, input_size) # batch size = 1\n",
    "\n",
    "# 在 no_grad 上下文下进行推理\n",
    "print(\"Performing inference with the loaded model...\")\n",
    "with torch.no_grad():\n",
    "    output = new_model(dummy_input)\n",
    "\n",
    "print(\"Inference successful.\")\n",
    "print(f\"Output shape: {output.shape}\")\n",
    "# 模型的输出现在是基于加载的参数计算得到的。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfb15049-c58e-4250-9ae3-a11d1c8effff",
   "metadata": {},
   "source": [
    "#### state_dict的严格模式与非严格模式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "302cbe99-82b7-49bb-a5e9-d1464d96683c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 定义 FNN 版本 1\n",
    "class SimpleFNN_v1(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(SimpleFNN_v1, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(hidden_size, output_size)\n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "# 定义 FNN 版本 2 (比版本 1 多一层)\n",
    "class SimpleFNN_v2(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, intermediate_size, output_size):\n",
    "        super(SimpleFNN_v2, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        self.relu1 = nn.ReLU()\n",
    "        # 注意这里的输出层大小与SimpleFNN_v1有所差别，即使stirct=False，\n",
    "        # 仍会导致加载失败，因为非严格模式并不能接受形状上的不兼容，所以需要加载前进行特殊处理。\n",
    "        self.fc2 = nn.Linear(hidden_size, intermediate_size) \n",
    "        self.relu2 = nn.ReLU()\n",
    "        self.fc3 = nn.Linear(intermediate_size, output_size) # 新增一层\n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu1(x)\n",
    "        x = self.fc2(x)\n",
    "        x = self.relu2(x)\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "\n",
    "# 模型参数\n",
    "input_size = 10\n",
    "hidden_size = 20\n",
    "intermediate_size = 15\n",
    "output_size = 3\n",
    "\n",
    "# 实例化并保存 v1 模型的 state_dict\n",
    "model_v1 = SimpleFNN_v1(input_size, hidden_size, output_size)\n",
    "v1_state_dict = model_v1.state_dict()\n",
    "\n",
    "# 实例化 v2 模型\n",
    "model_v2 = SimpleFNN_v2(input_size, hidden_size, intermediate_size, output_size)\n",
    "\n",
    "print(\"Loading V1 state_dict into V2 model with strict=False and checking return info...\")\n",
    "# 获取目标模型 (v2) 的 state_dict，用于检查形状\n",
    "v2_model_dict = model_v2.state_dict()\n",
    "\n",
    "# 创建一个新的 state_dict，只包含 v1 中存在且在 v2 中形状匹配的键值对\n",
    "# 这是解决 size mismatch 问题的关键\n",
    "filtered_state_dict = {k: v for k, v in v1_state_dict.items() if\n",
    "                       k in v2_model_dict and v.shape == v2_model_dict[k].shape}\n",
    "\n",
    "print(\"\\nFiltered state_dict keys to load (matching shapes):\")\n",
    "if filtered_state_dict:\n",
    "    for k in filtered_state_dict.keys():\n",
    "        print(f\"  {k}\")\n",
    "else:\n",
    "    print(\"  No keys with matching names and shapes found.\")\n",
    "\n",
    "# 使用 strict=False 加载过滤后的 state_dict，此时不会再有 size mismatch，因为我们只加载形状匹配的层。\n",
    "# strict=False 会忽略 v2 中存在但 filtered_state_dict 中不存在的键 (fc2, fc3)。\n",
    "try:\n",
    "    load_info = model_v2.load_state_dict(filtered_state_dict, strict=False)\n",
    "\n",
    "    print(\"\\nLoading completed successfully.\")\n",
    "    # load_info 会包含加载过程中被忽略的键 (missing_keys) 和 state_dict 中存在但在模型中不存在的键 (unexpected_keys)。\n",
    "    # 在这里，fc2 和 fc3 会是 missing_keys，因为它们在 model_v2 中，但不在 filtered_state_dict 中。\n",
    "    print(f\"Missing keys (exist in model_v2 but not in filtered_state_dict): {load_info.missing_keys}\")\n",
    "    # unexpected_keys 应该为空，因为我们已经过滤掉了所有不在 model_v2 中的键\n",
    "    print(f\"Unexpected keys (exist in filtered_state_dict but not in model_v2): {load_info.unexpected_keys}\")\n",
    "\n",
    "except Exception as e:\n",
    "    # 现在应该不会因为 size mismatch 而进入这个 except 块了\n",
    "    print(f\"An unexpected error occurred during loading: {e}\")\n",
    "    # 在异常发生时，load_info 可能未定义，这里为了安全，可以捕获一下或者在del前检查\n",
    "    load_info = None # 确保即使出错 load_info 也有一个值，避免 NameError\n",
    "\n",
    "# 清理\n",
    "del load_info, model_v1, model_v2, v1_state_dict, filtered_state_dict, v2_model_dict\n",
    "print(\"\\nCleanup complete.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e0f1562-e2b5-4e35-a6f4-34356df2bfa8",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### 保存和加载整个模型\n",
    "\n",
    "#### 模型整个模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ec94f60-fc19-4e0b-bd9b-be9117b7e99b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import os # 用于检查文件是否存在\n",
    "\n",
    "# 假设我们已经定义并实例化了一个 SimpleFNN 模型\n",
    "# （这里我们为了演示，只实例化）\n",
    "\n",
    "class SimpleFNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(SimpleFNN, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(hidden_size, output_size)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "# 定义模型参数\n",
    "input_size = 10\n",
    "hidden_size = 20\n",
    "output_size = 3\n",
    "\n",
    "# 实例化要保存的模型\n",
    "model_to_save = SimpleFNN(input_size, hidden_size, output_size)\n",
    "\n",
    "# 定义保存整个模型的文件路径\n",
    "entire_model_path = 'entire_simple_fnn_model.pth' # 不同的文件名以示区别\n",
    "\n",
    "# --- 保存整个模型实例 ---\n",
    "print(f\"Saving entire model instance to {entire_model_path}...\")\n",
    "try:\n",
    "    # 使用 torch.save() 保存整个模型对象\n",
    "    torch.save(model_to_save, entire_model_path)\n",
    "\n",
    "    print(\"Entire model instance saved successfully!\")\n",
    "    print(f\"Checking if file exists: {os.path.exists(entire_model_path)}\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"An error occurred while saving: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a3d1c28-9b26-461e-99eb-a54ee1577b5e",
   "metadata": {},
   "source": [
    "#### 加载整个模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73ccdeb9-98f7-4681-80de-b8947e7b2359",
   "metadata": {},
   "outputs": [],
   "source": [
    "# --- 加载整个模型实例 ---\n",
    "print(f\"\\nLoading entire model instance from {entire_model_path}...\")\n",
    "\n",
    "try:\n",
    "    # 使用torch.load()直接加载整个模型对象\n",
    "    # 注意：Pytorch 2.6版本之前，weights_only参数的默认值为False,但该版本之后的默认值为True，\n",
    "    # 因此，若要加载整个模型，需要手动将其设置为False。\n",
    "    loaded_model_instance = torch.load(entire_model_path, weights_only=False)\n",
    "\n",
    "    print(\"Entire model instance loaded successfully!\")\n",
    "    print(f\"Type of loaded object: {type(loaded_model_instance)}\")\n",
    "\n",
    "    # 您可以直接使用加载后的模型实例进行推理\n",
    "    print(\"\\nLoaded model instance created.\")\n",
    "\n",
    "    # 加载完成后，模型已经包含结构和参数\n",
    "\n",
    "    # 类似 state_dict 方法，加载用于推理的模型通常需要设置为评估模式\n",
    "    loaded_model_instance.eval()\n",
    "    print(\"Loaded model set to evaluation mode.\")\n",
    "\n",
    "    # 模拟输入进行推理\n",
    "    dummy_input = torch.randn(1, input_size)\n",
    "    with torch.no_grad():\n",
    "        output = loaded_model_instance(dummy_input)\n",
    "\n",
    "    print(\"Inference successful with loaded entire model.\")\n",
    "    print(f\"Inference output shape: {output.shape}\")\n",
    "\n",
    "\n",
    "except FileNotFoundError:\n",
    "    print(f\"Error: The file '{entire_model_path}' was not found.\")\n",
    "except Exception as e:\n",
    "    # !!! 注意：如果模型类定义不可用或不匹配，这里会抛出各种错误 !!!\n",
    "    print(f\"An error occurred while loading or inferring: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a9785d4-051a-4a8b-bcd1-9f781a895789",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### 保存和加载检查点文件示例\n",
    "\n",
    "#### 构建检查点字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0af13ce-9f2b-47ba-aac9-e8f4c22d3ef1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 假设我们有以下对象和变量\n",
    "# model: 当前训练中的模型实例\n",
    "# optimizer: 当前训练中的优化器实例\n",
    "# epoch: 当前完成的 epoch 数\n",
    "# step: 当前完成的总步数 (可选)\n",
    "# loss: 当前的训练 loss (可选)\n",
    "# best_metric: 历史最佳评估指标 (可选)\n",
    "# scheduler: 当前使用的学习率调度器实例 (如果使用的话)\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "# 示例：假设我们已经定义了模型、优化器，并且进行了一段训练\n",
    "class SimpleFNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(SimpleFNN, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(hidden_size, output_size)\n",
    "    def forward(self, x):\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "# 实例化模型和优化器 (这里只是为了构建 state_dict，不进行实际训练)\n",
    "input_size = 10\n",
    "hidden_size = 20\n",
    "output_size = 3\n",
    "lr=0.001\n",
    "model = SimpleFNN(input_size, hidden_size, output_size)\n",
    "optimizer = optim.Adam(model.parameters(), lr=lr)\n",
    "\n",
    "# 模拟一些训练状态\n",
    "current_epoch = 5\n",
    "current_step = 500\n",
    "current_loss = 0.15\n",
    "best_validation_accuracy = 0.88\n",
    "\n",
    "# 如果使用了学习率调度器，也获取其状态字典\n",
    "# scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)\n",
    "# scheduler_state = scheduler.state_dict() # 需要先定义和初始化 scheduler\n",
    "\n",
    "# --- 构建检查点字典 ---\n",
    "checkpoint = {\n",
    "    'epoch': current_epoch,\n",
    "    'step': current_step,\n",
    "    'model_state_dict': model.state_dict(),\n",
    "    'optimizer_state_dict': optimizer.state_dict(),\n",
    "    'loss': current_loss,\n",
    "    'best_accuracy': best_validation_accuracy,\n",
    "    # 'scheduler_state_dict': scheduler_state, # 如果使用调度器则包含此项\n",
    "    # 'torch_rng_state': torch.get_rng_state(), # 如果需要严格复现，保存 RNG 状态\n",
    "    # 'numpy_rng_state': np.random.get_state(), # 如果使用 NumPy，保存 RNG 状态\n",
    "    # 'random_rng_state': random.getstate(),  # 如果使用 Python random，保存 RNG 状态\n",
    "}\n",
    "\n",
    "print(\"Checkpoint dictionary built:\")\n",
    "print(checkpoint.keys())\n",
    "# 打印其中一个 state_dict 的键来确认\n",
    "print(\"\\nKeys in model_state_dict within checkpoint:\")\n",
    "print(checkpoint['model_state_dict'].keys())\n",
    "\n",
    "# 注意：如果模型和优化器没有进行过实际的训练步骤，它们的 state_dict 可能不完整（特别是优化器）\n",
    "# 在实际训练中，确保在保存检查点时，模型和优化器已经进行了至少一步更新。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72abb967-3697-4039-ae5b-59836549fab4",
   "metadata": {},
   "source": [
    "#### 保存检查点文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8939459-eca9-41cd-9de8-c562488b82fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 继续上面的例子...\n",
    "\n",
    "# 定义检查点文件保存路径\n",
    "checkpoint_path = 'my_training_checkpoint.pth' # 或者根据 epoch/性能命名\n",
    "\n",
    "# --- 保存检查点字典 ---\n",
    "print(f\"\\n保存检查点信息至文件 {checkpoint_path} 中...\")\n",
    "try:\n",
    "    torch.save(checkpoint, checkpoint_path)\n",
    "    print(\"检查点文件保存成功!\")\n",
    "    print(f\"检查文件是否存在: {os.path.exists(checkpoint_path)}\")\n",
    "\n",
    "except Exception as e:\n",
    "    print(f\"An error occurred while saving checkpoint: {e}\")\n",
    "\n",
    "# 清理\n",
    "# del model, optimizer, checkpoint # 在实际脚本中，这些通常会一直存在"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2649bf14-d7da-42d1-83f9-8f6062d19b74",
   "metadata": {},
   "source": [
    "#### 从检查点恢复模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "625917d0-45de-4d40-950b-05295b75d9cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 继续上面的例子...\n",
    "\n",
    "# 定义检查点文件路径 (假设它已经存在)\n",
    "checkpoint_path = 'my_training_checkpoint.pth'\n",
    "\n",
    "# --- 加载检查点以恢复训练 ---\n",
    "print(f\"\\n正在从检查点文件 {checkpoint_path} 开始恢复训练...\")\n",
    "\n",
    "# 确定加载设备\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(f\"Loading onto device: {device}\")\n",
    "\n",
    "try:\n",
    "    # 1. 加载检查点字典，并指定映射设备\n",
    "    checkpoint = torch.load(checkpoint_path, map_location=device)\n",
    "    print(\"检查点字典装载完成...\")\n",
    "\n",
    "    # 2. 实例化模型 (结构必须与保存时一致)\n",
    "    loaded_model = SimpleFNN(input_size, hidden_size, output_size)\n",
    "    # 将模型移动到目标设备\n",
    "    loaded_model.to(device)\n",
    "    print(\"创建新模型并移动到指定的设备...\")\n",
    "\n",
    "    # 3. 实例化优化器 (类型和配置应与保存时一致)\n",
    "    # !!! 使用加载参数后的模型的参数初始化优化器 !!!\n",
    "    loaded_optimizer = optim.Adam(loaded_model.parameters(), lr=lr) # 初始 lr 会被 state_dict 覆盖\n",
    "    print(\"创建新的优化器实例...\")\n",
    "\n",
    "    # 4. 加载模型状态\n",
    "    loaded_model.load_state_dict(checkpoint['model_state_dict'])\n",
    "    print(\"从检查点文件加载模型的state_dict...\")\n",
    "\n",
    "    # 5. 加载优化器状态\n",
    "    loaded_optimizer.load_state_dict(checkpoint['optimizer_state_dict'])\n",
    "    print(\"从检查点文件加载优化器的state_dict...\")\n",
    "\n",
    "    # 6. 恢复其他状态\n",
    "    start_epoch = checkpoint['epoch'] + 1 # 从下一个 epoch 开始\n",
    "    recovered_step = checkpoint['step']\n",
    "    recovered_loss = checkpoint['loss']\n",
    "    recovered_best_accuracy = checkpoint['best_accuracy']\n",
    "\n",
    "    # 如果使用了调度器\n",
    "    # loaded_scheduler = optim.lr_scheduler.StepLR(loaded_optimizer, step_size=30, gamma=0.1)\n",
    "    # loaded_scheduler.load_state_dict(checkpoint['scheduler_state_dict'])\n",
    "    # print(\"Scheduler state loaded.\")\n",
    "\n",
    "    # 如果保存了 RNG 状态并需要严格复现\n",
    "    # torch.set_rng_state(checkpoint['torch_rng_state'])\n",
    "    # np.random.set_state(checkpoint['numpy_rng_state'])\n",
    "    # random.setstate(checkpoint['random_rng_state'])\n",
    "    # print(\"RNG states restored.\")\n",
    "\n",
    "    print(f\"\\n从 Epoch {start_epoch}和 Step {recovered_step} 继续进行训练...\")\n",
    "    print(f\"Recovered Loss: {recovered_loss}, Best Accuracy: {recovered_best_accuracy}\")\n",
    "\n",
    "\n",
    "    # 7. 设置模型模式 (恢复训练则设置为 train())\n",
    "    loaded_model.train() # 设置为训练模式\n",
    "    print(\"将模型设置为训练模式...\")\n",
    "\n",
    "except FileNotFoundError:\n",
    "    print(f\"Error: Checkpoint file not found at {checkpoint_path}. Starting training from scratch.\")\n",
    "    # 在实际应用中，如果文件不存在，你会选择从头开始训练\n",
    "    # start_epoch = 0\n",
    "    # loaded_model = SimpleFNN(...)\n",
    "    # loaded_optimizer = optim.Adam(...)\n",
    "    # etc.\n",
    "except Exception as e:\n",
    "     print(f\"An unexpected error occurred during loading checkpoint: {e}\")\n",
    "     # 根据需要处理加载失败的情况\n",
    "finally:\n",
    "     # 清理生成的文件 (可选)\n",
    "    if os.path.exists(checkpoint_path):\n",
    "        os.remove(checkpoint_path)\n",
    "        print(f\"\\n清理检查点文件：{checkpoint_path}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
